Factors affecting the performance of maternal health care providers in Armenia.

BACKGROUND: Over the last five years, international development organizations began to modify and adapt the conventional Performance Improvement Model for use in low-resource settings. This model outlines the five key factors believed to influence performance outcomes: job expectations, performance feedback, environment and tools, motivation and incentives, and knowledge and skills. Each of these factors should be supplied by the organization in which the provider works, and thus, organizational support is considered as an overarching element for analysis. Little research, domestically or internationally, has been conducted on the actual effects of each of the factors on performance outcomes and most PI practitioners assume that all the factors are needed in order for performance to improve. This study presents a unique exploration of how the factors, individually as well as in combination, affect the performance of primary reproductive health providers (nurse-midwives) in two regions of Armenia.
METHODS: Two hundred and eighty-five nurses and midwives were observed conducting real or simulated antenatal and postpartum/neonatal care services and interviewed about the presence or absence of the performance factors within their work environment. Results were analyzed to compare average performance with the existence or absence of the factors; then, multiple regression analysis was conducted with the merged datasets to obtain the best models of "predictors" of performance within each clinical service.
RESULTS: Baseline results revealed that performance was sub-standard in several areas and several performance factors were deficient or nonexistent. The multivariate analysis showed that (a) training in the use of the clinic tools; and (b) receiving recognition from the employer or the client/community, are factors strongly associated with performance, followed by (c) receiving performance feedback in postpartum care. Other - extraneous - variables such as the facility type (antenatal care) and whether observation was on simulated vs. real patients (postpartum care) also had a role in observed performance.
CONCLUSION: This study concludes that the antenatal and postpartum care performance of health providers in Armenia is strongly associated with having the practical knowledge and skills to use everyday tools of the trade and with receiving recognition for their work, as well as having performance feedback. The paper recognized several limitations and expects further studies will illuminate this important topic further.

Background

Training and service delivery organizations have tried for decades to improve the quality and access of healthcare services in developing countries by providing training to health care providers. The assumption has been that the gap in providers' performance is attributable to inadequate knowledge and skills – and therefore, training is what is called for. Millions of training dollars later, healthcare indicators are little improved and providers are still in need of support [1].

Realizing that training is oftentimes not the only solution, organizations such as IntraHealth International have searched for a key to improving outcomes at the most basic point of contact between the provider and the client. IntraHealth analyzed both domestic research in Human Performance Technology (HPT), as well as the current research on the systems influencing family planning provider performance within developing countries themselves. A line of research on family planning provider performance has focused on capturing a holistic view of the entire family planning program in a given region or country, where data is collected from experts in the area [2]. Although this research has offered evidence of overall program effects on population-level outcomes, it is of limited use in understanding specific conditions affecting the performance of providers at the facility level.

HPT experts in domestic countries, on the other hand, have long established that optimal worker performance is predicated on supporting the worker in a variety of areas, including: (1) clear job expectations; (2) timely performance feedback; (3) adequate environment and tools; (4) internal motivation and/or external incentives; (5) knowledge and skills; (6) capacity to do the job. IntraHealth/PRIME II adapted this list of factors to the international development field, subtracting 'capacity' and adding 'organizational support' to address inadequacies in the other five areas, such as supervision to deliver feedback and clarify expectations for providers. With these premises at hand, organizations have recently developed and utilized the Performance Improvement (PI) approach to improve provider performance and project interventions in the developing world [3-8].

Although the logic of looking at workers' overall environment when analyzing productivity has been widely documented [9], there is little empirical research exploring the relationships between worker performance and the performance factors, and even less in developing or newly-independent countries. One of a few domestic studies was carried out by Hwang using a mail survey among employees and supervisors at the Office of the Inspector General in the Florida Department of Children and Families. Using an elaborate framework of five components of performance (quantity, quality, efficiency, problem-solving capacity and adaptability) and seven performance factors (performance specification, capacities, knowledge and skills, job/task design, incentives, feedback and resources and tools), he finds that in the employees' questionnaire, feedback is related to performance, in particular the efficiency dimension of performance. Though there are relationships between the factors and performance, in the questionnaires filled out by supervisors, results are less definite. Among the limitations of the study is that its cross-sectional design precluded making cause-effect inferences [10].

This study uses a simple framework: it assumes that human performance is facilitated and/or hindered by the 'performance factors' listed above and hypothesizes that not all factors are equal in their effects on performance. The objective of the study is to determine which factor(s) have a higher association with provider performance (See study framework in Figure 1). In order to control for extraneous factors, all providers are of similar cadre (i.e. nurse-midwives) and working in the same institution (MOH). Also, the type of establishment is limited to the primary/intermediate level of care (which is the PRIME project mandate), namely polyclinics, health centers and posts. Although human performance is comprised of both behavior and its accomplishments [11], for this study only the behavioral component of performance is investigated and will be determined through observation of health worker's realization of job tasks.

Figure 1

Framework for the performance factors and provider performance (Control variables: provider and clinic characteristics)

Methods

The study was conducted in Armenia as an extension to a baseline assessment of provider performance to inform the implementation of a USAID-funded project to improve maternal and neonatal health. It constituted a facility-based survey drawing on a sample of nurses and midwives working at three types of service delivery points: polyclinics, health centers and primary posts or FAPs, in the regions of Lori and Shirak. Consenting health care providers and clients were observed while providing two kinds of reproductive health services (prenatal and postpartum care); trained clinical data collectors recorded the completion of clinical and non-clinical tasks using a checklist derived from the MEASURE Evaluation's Quick Investigation of Quality (QIQ) tool [12]. Because it was not always possible to observe a real client visit (the total fertility rate in Armenia is a low 1.7 live births/woman – DHS 2000) [13], providers were sometimes asked to simulate a situation with a second interviewer standing in for the client (analyses will take this into account). After the provider was observed, she was interviewed about her work environment and the other performance factors.

Using the StatCalc feature of Epi Info 6.04, the desired sample size was estimated to be 300 based on a total provider population of 3000; an expected frequency of 40% (average performance for clinical skills); a worst acceptable error of 15%; a confidence level of 95%; and a 10% margin for loss of data. Given the high numbers required, all nurse-midwives working at outpatient facilities in both Lori and Shirak regions were selected for interview. The final number of providers observed and interviewed was 285.

A total of 11 two-person teams, an observer (physician) and an interviewer (non-clinician), collected data in the field. The clinician carried out the observations while the non-clinicians administered the performance factors questionnaires. Data collectors participated in a weeklong training on proper methods of completing questionnaires and checklists. Study instruments had been reviewed by local clinical experts, translated into Armenian, and pre-tested twice for consistency and comprehensibility.

Field work took place simultaneously in Lori and Shirak regions. An average of two providers was observed/ interviewed each day per data collector team spending a total of 13 working days to complete observations/interviews during August 2002. Some providers, especially at the FAP level, were visited multiple times to complete the interviews, as they were not found at their post. Supervisors ensured correct application and completion of instruments and a field coordinator organized all logistics and performed quality control of the field work.

The average performance score for each provider was compared to the dichotomous answers received from the performance factor questionnaire. The resulting significant factors were entered into a stepwise multiple regression model to discover which factor(s) would become better overall predictors of performance. Some background variables, such as clinic type, provider's age and years working at the facility were added to the model.

Results

Provider characteristics

All respondents but one were female and 82% were married at the time of the interview. With a median age of 42, they had worked in reproductive health on average for 20 years, with 14 years on average at the interview facility. These variables are dichotomized for the bivariate analysis for ease of interpretation, but were run as originally (continuous) in the multivariate analysis.

Prenatal care skills

The prenatal care checklist included 42 tasks to perform during each client visit and all providers were expected to fulfill them. Performance was substandard across the board (an average score of 14.2, or 38% of the total possible score) with acutely low scores in several triage and clinical areas such as "Washes hands with soap and water and dries them" and "Takes temperature" and counseling areas such as "Informs woman of positive and side effects of medicines during pregnancy" (see Table 1). Though there was a slight difference in performance scores between real (14.9) and simulated (13.5) client-provider interactions, such difference did not reach statistical significance (p = 0.07).

Table 1

Percentage of prenatal care providers who fulfilled each skill item, and average scores

#	ITEM	Nurses / midwives (285)
1.	Washes hands with soap & water and dries them	13.0 (284)
2.	Greets and calls woman by her first name and introduces him/herself if first visit	91.6
3.	Ensures woman is in a comfortable environment	51.2
4.	Explains purpose of the session and nature of the procedures	53.5 (284)
5.	Asks questions and allows client to express herself	88.4 (284)
6.	Pays attention and is interested in personal problems of the woman	68.1
7.	Reviews clinic record before start of sesson/does new record for new client	63.9
8.	In case it's possible, performs medical tests (urine, blood)	53.4 (279)
9.	Explores pulse rate	20.8 (283)
10.	Explores blood pressure	91.9
11.	Takes temperature	13.0 (284)
12.	Gets anthropometric measurements: weight, height	48.2 (284)
13.	Examines skin and conjunctivae	16.1
14.	Checks for oedema, redness and varicose veins – legs	44.9
15.	Examines thyroid, mouth	4.2
16.	Examines breasts	31.0 (284)
17.	Examines the heart and lungs, if necessary send her to the relevant specialist	13.7
18.	Inspects and palpates abdomen for scars, pigmentation...	11.6
19.	Palpates uterus and performs maneuvers to detect fetal position and situation	29.5 (281)
20.	Measures uterine height, abdomen circumference and listens to the fetal heart rate (in case of pregnancy ≥ 18 weeks)	37.7 (281)
21.	Determines weeks of pregnancy and probable delivery date	38.9
22.	Informs woman about the progress of pregnancy	29.8
23.	Informs woman about her health condition	30.5
24.	Informs woman about the fetus' health condition	16.5
25.	Informs woman about any complications	29.8
26.	Orients woman on the place of delivery (hospital contacts, transportation, etc.)	46.3
27.	Orients woman about management of common pregnancy-related afflictions	33.7
28.	Orients woman about personal hygiene, rest and general care	69.1
29.	Orients woman about gender, sexuality and STI prevention	15.1
30.	Orients woman about alarm signs: pain, fever, bleeding and loss of vaginal fluid	34.4
31.	Counsels woman about her nutritional needs and prescribes iron and folates	16.8
32.	Informs woman of positive and side effects of medicines during pregnancy	6.3
33.	Orients woman about breast feeding, baby vaccination and use of contraception	37.3 (284)
34.	Solicits questions to ensure client has understood	26.3
35.	Schedules appointment according to clinic needs and woman's convenience	54.4
36.	Records all findings, assessments, diagnosis and care with client	38.9
37.	Thanks client for her time	50.5
AVERAGE PERCENTAGE SCORE		38.4%
TOTAL SKILLS SCORE (Items 0 – 37)2		14.2

1 Percentages of total valid observations 2 Obtained by adding up all the positive answers to each item: range 0 – 37 (0 = Nil; 37 = All) * p < 0.05; ** p < 0.01; NS: not significant

Postpartum care skills

The postpartum skills checklist included 37 tasks that nurse-midwives had to perform during the client visit. Again, performance was substandard, although slightly higher than the prenatal care results (average score of 16.4, or 51% of the total possible score). Again, there were weak clinical areas, such as examining the skin and conjunctivae as well as for swelling and varicosity in legs. Low scores are also seen in preventive areas such as orienting the woman on sexuality, follow up services and contraception (See Table 2). Unfortunately, only 23% of the 278 observations were done in real patients and the performance observed here was significantly higher (score of 19.2) than with simulated observations (score of 15.7).

Table 2

Percentage of postpartum care providers who fulfilled each skill item, and average scores

#	ITEM	Nurses / midwives (285)
1.	Washes hands with soap & water and dries them	22.8
2.	Greets and calls woman by her first name and introduces him/herself if first visit	91.9
3.	Ensures woman is in a comfortable environment	38.0 (284)
4.	Explains purpose of the session and nature of the procedures	52.5 (284)
5.	Asks questions and allows client to express herself	85.6 (284)
6.	Pays attention and is interested in personal problems of the woman	69.5
7.	Asks about last pregnancy and delivery: evolution, outcome, any complications	71.9
8.	Asks about present status and any danger signs	73.7
9.	Explores pulse rate	20.9 (282)
10.	Explores blood pressure	66.3
11.	Takes temperature	58.1 (284)
12.	Examines skin and conjunctivae	18.7 (284)
13.	Checks for oedema, redness and varicose veins – legs	16.2 (284)
14.	Inspects and palpates abdomen for uterine involution	41.8
15.	Examines breasts and inquires for any lactation problem	74.7
16.	Examines lochia (amount, color, smell)	48.4
17.	Asks about baby's health: sleeping, feeding, posture, skin color, breathing, fever	68.1
18.	Assesses baby's health: feeding, posture, skin color, breathing, fever	57.5
19.	Informs woman about her health condition	44.9
20.	Informs woman about the baby's health condition	49.8
21.	Informs woman about potential complications and trains on self assessment	40.0
22.	Orients woman about breast feeding and breast care	86.3
23.	Orients woman about personal hygiene	74.0
24.	Orients woman about gender, sexuality and STI prevention	24.2
25.	Counsels woman about her nutritional needs	60.4 (283)
26.	Orients woman about hospital/clinic services (e.g. location, hours, etc.) for follow-up	32.4 (284)
27.	Orients woman about baby vaccination	56.1
28.	Orients woman about birth spacing and contraception	19.4 (284)
29.	Solicits questions to ensure client has understood	31.6
30.	Schedules appointment according to clinic needs and woman's convenience	60.4
31.	Records all findings, assessments, diagnosis and care with client	39.3
32.	Thanks client for her time	46.3
AVERAGE PERCENTAGE SCORE		51.3%
TOTAL SKILLS SCORE (Items 0 – 32)2		16.4

1 Percentages of total valid observations 2 Obtained by adding up all the positive answers to each item: range 0 – 32 (0 = Nil; 32 = All)

The performance factors: Are they present or absent?

Clear job expectations

Almost 7 of 10 providers lacked job description; when asked how they knew what to do for their jobs, 69% answered "through oral explanation from the supervisor or other person." When providers were asked whether standards for their performance had been set (i.e. they were told how they should do their job), 78% responded affirmatively; in further questioning, 37% stated having guidelines, 35% other written material, and 21% had protocols.

Motivation and incentives

Asked whether providers received bonuses or raises for good performance, 92% of nurses/midwives responded negatively. Asked what types of non-monetary incentives they received, providers working in prenatal care (clinic-based) responded: "verbal recognition from supervisors" (44.3%), followed by "training courses" (21.3%), and "free/reduced medicine" (14.6%). Providers offering postpartum care (community-based) responded "verbal recognition from clients or the community" (36.3%), closely followed by "respect in the community" (31%), "in-kind products" (e.g. pack of coffee, chocolate bar) (19 %), and "services in return" (e.g. cutting firewood, farming, etc.) (11.4 %).

Feedback on performance

A large majority of nurses/midwives (95%) stated they had received "feedback about [their] job performance." When probed on the characteristics of the feedback, nearly all answers considered it work-related (99.6 %), related to standards and not to behavior (93.6%), immediate and frequent enough to help remember what they did (92.8%), selective and specific (92.5%), and positive and constructive (93.2%). When asked for an example of the feedback, however, nearly two-thirds cited occasions when a supervisor or a doctor had praised them for some good deed, indicating that what was considered to be feedback could also be considered verbal recognition. When asked from whom they received the "feedback", 600 answers listed a range of sources, from clients and colleagues, to the mayor's office.

Organizational support

When asked if they had received performance reviews from supervisors, 64% said they had; in describing the reviews, more than 80% said they were oral and informal exercises.

Supervision visits occur frequently, according to respondents: 94% of providers had received a supervisory visit during the previous 6 months. Visits often lasted over 2 hours, with physicians providing services such as specialty consultations or vaccinations. Though supervision was often and quite long, 86% of providers characterized supervisors' tasks as administrative (e.g. "checks forms, vaccinations") and 24% as clinical ("sees patients and works in the clinic"). In less than 10% of responses, the supervisor was said to have provided feedback on performance. This finding is perhaps explained by administrative visits by nurses and occasional clinical consultations by visiting physicians.

Environment (tools and equipment) and work organization

Asked if their workplace was adequate, 76% said "yes", although only 40% said it was "comfortable". Nearly 60% of providers said they did not have the tools "to do the job well." When asked what equipment was needed, First Aid medicines (13.5%), surgical instruments (10.5%) and scales (9.4%) were listed, averaging 2.8 items mentioned per individual.

As a follow-up question on the availability of equipment and tools, providers were asked if they had been trained in the use of the clinic tools – nearly 75% said they had been.

The last area explored was whether the provider was satisfied with the way the work was organized. Most respondents (75%) answered "yes" to this question.

Knowledge and skills

Forty percent of providers had not received any training in reproductive health. Of those trained, about one third had attended a course in 2002, with the majority attending in 2000. Most providers (82%) believed they had the necessary knowledge and skills to do their job and 97% of them claimed to be able to apply what they had learned to their work.

Relationships between performance factors and actual performance

Among the 19 variables representing all the performance factors, 11 were found to have a significant relationship to performance (see Table 3). The most significant are:

Table 3

Mean prenatal and postpartum care scores by presence/absence of performance factors

VARIABLE/FACTOR	RESPONSE (%)	PNC MEAN SCORE	N	PPC MEAN SCORE
ANALYSIS TYPE
Type of observation	Real	14.9	135	19.2** (N= 65)
	Simulated	13.5	150	15.7 (N = 213)
BACKGROUND
1. Facility type	Polyclinic/Women's consultation	17.4**	64	18.0
	Ambulatory/Health Center	13.4	68	15.7
	FAP	13.2	153	16.2
2. Worker category	Nurse	13.5*	170	16.9
	Midwife – Nurse/midwife	15.4	108	16.0
3. Age	≤ 41 yrs.	13.9	140	16.1
	42+	14.4	145	16.8
4. Years working in the facility	0–11	14.1	138	16.4
	12+	14.3	147	16.5
JOB EXPECTATIONS
5. Has job description	Yes	15.7*	76	18.8**
	No, DK	13.6	209	15.6
6. Whether standards for performance have been set	Yes	14.4	223	16.8
	No	13.4	62	15.3
MOTIVATION & INCENTIVES
7. Receive bonuses or raises for good work	Yes	14.0	23	16.0
	No	14.2	262	16.5
8. Non-monetary incentives (employer) – 1streply	Yes	14.4**	272	16.7**
	No	8.6	13	10.7
9. Non-monetary incentives (community) – 1streply	Yes	14.3*	280	16.6**
	No	7.6	5	7.4
10. Opportunities for promotion	Yes	15.8	48	17.7
	No	13.8	220	16.2
11. Disincentives for job badly done	Yes	14.2	112	16.3
	No	14.8	127	15.9
FEEDBACK
12. Receive feedback about job performance	Yes	14.3	265	16.4
	No	12.7	15	16.3
13. Appropriate Feedback (composite)	Yes	14.6+	217	16.9#
	No, DK	12.8	68	15.1
14. Having Performance Reviews	Yes	14.8*	183	17.4**
	No, DK	13.1	102	14.8
ORGANIZATIONAL SUPPORT
15. Received supervision in last 6 months	Yes	14.4*	267	16.6
	No	10.9	18	14.2
16. Nature of supervision – 1st reply	Admin	14.8	213	16.8
	Other	11.8	35	16.1
WORK ORGANIZATION & ENVIRONMENT
17. Adequacy of Job place (composite)	Yes (≥ 75%)	14.3	184	16.5
	No (< 75%)	13.9	101	16.4
18. Has the necessary equipment, instruments and supplies	Yes	15.5**	117	17.0
	No	13.3	166	16.1
19. Satisfied w/organization of work	Yes	14.9**	218	16.9*
	No, DK	11.8	67	14.9
KNOWLEDGE & SKILLS
20. Believes has necessary skills to do the job	Yes	14.8**	234	17.0**
	No, DK	11.5	51	13.9
21. Whether received training in RH	Yes	15.0*	171	17.1*
	No	13.0	114	15.4
22. Been trained in the use of tools	Yes	15.0**	211	17.5**
	No	11.8	72	13.3
23. Year of last training	2001–2002	15.3	128	17.7
	1972–1999	13.9	43	17.0
OVERALL MEAN SCORE		14.2	285	16.4

* p < 0.05; ** p < 0.01; + p = 0.047; # p = 0.05 *variables significant for both clinical areas are highlighted

- Having a job description

- Receiving non-monetary incentives, from the employer and from the community; receiving bonuses or raises, opportunities for promotion and disincentives are found not to be related to performance.

- Being satisfied with the organization of the work;

- Having the necessary equipment, instruments and supplies was only critical to performance for prenatal care.

- Three of four variables in knowledge and skills were found to be significant with prenatal and postpartum care performance: if the provider believes [he/she] has the necessary skills to do the job, whether they have received training in RH and having been trained in the use of clinic tools appear significantly related to all scores.

- Receiving performance reviews

Appropriate feedback does not reach significance by close margin (p = 0.047 and p = 0.05) while the more direct question on whether the provider receives feedback about his/her performance was not found to be influential.

Whether the provider received supervision in the last 6 months was related to performance only in the area of prenatal care. The kind of supervision, on the other hand, produced some small differences only in the area of prenatal care, which did not reach statistical significance.

Multivariate analysis

Variables significantly associated with performance at the bivariate level plus available background characteristics were selected as independent variables in multiple regression analyses of performance (the dependent variable) – see Table 2. Analyses were done separately for prenatal and postpartum care performance.

Prenatal care and performance

Eleven of the 21 performance factor variables had a significant relationship with prenatal care performance and were included in the linear regression analysis, as well as the four background variables of importance (age, years working in the facility, type of worker and facility type), despite the lack of association in the first two. Results appear in Table 4.

Table 4

Features of the best-fit multiple regression model of performance (prenatal care) and factors

Variables in the model	Unstandardized Coefficients – B	Std Error	Standardized Coefficients – Beta	t	Sig.
(Constant)	11.811	2.211	--	5.342	.000
Incentives by employer	4.687	1.880	.144	2.493	.013
Have you been trained in using the tools	2.317	.910	.149	2.547	.011
Facility type	-1.659	.479	-.200	-3.466	.001

Dependent Variable: Provider Performance Score R = 0.319; R Square = 0.102; R Square Adjusted = 0.092 Variables excluded: Type of Worker, Years working in the facility, Age of provider, Having a job description, Whether there's feedback, Whether there's performance reviews, Receiving incentives from the community, Having the necessary equipment, Whether satisfied with work, Whether thinks has the necessary knowledge and skills to do job, Whether received supervision, Whether received training, Scenario (real vs. simulated).

The table shows that 3 of the 15 variables entered into the regression equation became best predictors of performance in this clinical area. These are, in order of importance (based on the standardized -Beta coefficients), the type of clinic the provider works in, whether the provider [has] been trained in using the tools and whether the provider receives non-monetary incentives from the employer. The adjusted R Square for the model is 0.09.

Postpartum care and performance

There were nine significant performance factors related to postpartum care performance at the bivariate level. These and the four background variables (despite not showing associations) were included in a stepwise multiple regression as before. Type of observation was also included because of its significant relationship with performance. Table 5 presents the results obtained.

Table 5

Features of the best-fit multiple regression model of performance (postpartum care) and factors

Variables in the model	Unstandardized Coefficients – B	Std. Error	Standardized Coefficients – Beta	t	Sig.
(Constant)	4.720	3.2,868	---	1.646	.101
Incentives by community	7.008	2.874	.137	2.438	.015
Have you been trained in using the tools	3.821	.861	.248	4.438	.000
Scenario (real vs. simulated)	3.113	.885	.196	3.516	.001
Are there any performance reviews?	1.997	.787	.143	2.539	.012

Dependent Variable: Provider Performance Score R = 0.403; R Square = 0.162; R Square Adjusted = 0.15 Variables excluded: Worker type, Facility type, Years working in the facility, Age of the provider, Having a job description, Whether received feedback, Having received incentives from employer, Having the necessary equipment, Whether satisfied with work, Thinks having the necessary knowledge and skills for the job, Whether received supervision, Whether received training.

This time, the regression analysis yielded four predictor variables. Using the Beta coefficient as a measure of relative order, [has] been trained in using the tools is the strongest predictor for postpartum care, followed by whether the scenario was real vs. simulated. A third variable of significance, having received performance reviews, is followed closely by receiving non-monetary incentives from the community. The adjusted R Square for this model is 0.15.

Discussion

The first important finding of the study is the statistical significance of relationships between performance and the performance factors. On the one-to-one relationships, 11 of the 21 variables tested had a significant effect on performance and more importantly, 7 of the 11 variables were the same for both clinical areas. They are:

Job expectations

1. having a job description,

2. having had performance reviews,

Motivation/incentives

3. receiving non-monetary incentives (from the employer),

4. receiving non-monetary incentives (from the community),

5. having been trained in the use of tools [for the job],

6. believing to have the necessary skills for the job, and

7. Having received training in RH.

A second key finding is that out of bivariate and multivariate analyses the performance factors seem to have more weight on performance than critical background conditions such as age of the worker or years working in the facility (see Table 6).

Table 6

Predictors of performance in PNC and PPC skills areas and order of importance

Predictors	PNC	PPC
Analysis Type
Real vs. simulated observation		2nd
Background
Type of facility in which provider works	1st
Performance Factors
Motivation & Incentives
Incentives by employer	3rd
Incentives by community		4th
Knowledge & Skills
Having been trained in the use of clinic tools	2nd	1st
Feedback (& Org. Support)
Having had performance reviews		3rd
R Square (adjusted)	0.09	0.15

Additionally, it is clear that for prenatal care, the type of facility in which the provider works is definitely associated with performance. This same variable does not exert any influence over performance in postpartum care. This seems to be largely due to the fact that nurses and midwives who work at the higher polyclinic level provide more clinical services than those at FAPs. They also receive better support in supervision and facility maintenance. The same does not hold true for postpartum care. Most providers working in FAPs contact and refer pregnant women to higher level clinics for service. Training in the use of clinic tools and equipment and verbal recognition by employer or community are predictors for both areas of performance – quite a finding for the study. For postpartum care performance, having performance reviews also predicts performance. The type of observation (real vs. simulated) also affected the outcome variable for this clinical area.

Conclusions

Although further studies of this nature are needed to confirm the relative importance of factors, in this study it is interesting to find that a number of aspects in the providers' work environment (as perceived by them) do correlate with observed performance. The multivariate analysis further determines a subset of factors and variables that seem more critical to performance. For example, results seem to indicate is not so much the theoretical knowledge but the practical application of skills in the use of everyday clinic tools that associates with improved performance. This finding seems to reinforce the need for 'essential learning' approaches for improved performance that several training organizations, including IntraHealth, are developing for use in developing countries.

Another predictor of importance is receiving non-monetary incentives by either providers' employers or the community for whom they work. In a context of economic hardship affecting fair and prompt payment of salaries to health workers such as in Armenia, non-monetary incentives in the form of recognition, in-kind contributions, community respect and assistance with services can become powerful motivators to enhance performance. The difference between employer's and community incentives between prenatal and postpartum care seems to indicate where these services occur mostly, prenatal care facilities level and postpartum care at the community level.

For postpartum care, having a performance review from a supervisor as a predictor of performance suggests that this mechanism is most important when people work in less structured environments, often working away from the clinic or the office.

Limitations

This study is largely of exploratory nature. As mentioned, there have been no empiric studies that we know of combining perceived performance factors and observed performance in the field of FP/RH in low-resource settings. Hence, interpretation of findings needs to be done with caution. Several caveats to the study should be mentioned.

First, the cross-sectional nature of the study, and corresponding associations found, do not imply causal relationships between the factors and provider performance. Though questions related to performance factors were asked with a retrospective connotation (e.g. "in the past 6 months, have you received any supervisory visit?"), memory recall and rationalization may curtail providers' recollection of factors affecting their performance. The multivariate analyses has been carried out by merging data from two different exercises and data collectors, thus arguably adding variation (and a degree of error) to the usual limitations.

Another aspect worth mentioning is the inherent challenges in the application of the factors questionnaire by interviewers. Though pre-tested and improved through pilot testing, several concepts with heavy Western connotations (e.g. what providers answered as "feedback" seems to have been other types of interaction).

We have also used a "proxy" of performance, through the observation of skills. There are other components of performance (e.g. accomplishments) not included in the study. The low R square rates obtained implies our model has captured only a small percentage of the total variation, suggesting that there might be several other factors associated with performance or perhaps better instruments/methods to ascertain it.

In addition, there is an added bias associated with the "Hawthorne effect" in both direct observation and "hypothetical client" approaches to measuring provider performance. Although these approaches are widely recognized as having inherent limitations, both are considered among the best methods for assessing a provider's clinical behavior in a truly technical and systematic way [14,15]. One such study, however, directly stated that the data collected through direct observation using the QIQ tool, which was utilized in the study, is as reliable as alternative methods for data collection, such as client exit interviews and "mystery clients" [16]. For our purposes, however, where we were assessing a provider's technical performance, direct observation was key for obtaining reliable data.

Furthermore, there is a perceived loss of complexity within the provider-client interaction when a "hypothetical client" is used in lieu of a normal client, as was used in some of the data collection. Case simulation might not replicate entirely the quality and completeness of the normal exchange occurring during direct observation of a provider with a real client, thus producing lower performance scores and possibly affecting the appearance of performance factors in unknown ways [17]. However, once again, "hypothetical clients" have been widely recognized as a reliable and systematic way to assess technical competency when other alternatives are not viable due to low case load for years [18]. In our case, though unavoidable, differences did appear in our measures of performance for postpartum care between real and hypothetical scenarios.

In all, this study has demonstrated the feasibility of the model. As applied, performance was associated mostly with workers' perceptions of having the practical knowledge of tools existing in the facility and of being acknowledged by employers or clients, two important factors to have in mind when considering quality and productivity of work. Further refinement of observation and interview tools are needed. Replication of this study (planned for Nigeria and Bolivia) should shed light about the relative importance of factors in other contexts. Their results can assist development efforts involving human resources by focusing on priority areas, and save valuable financial resources.

List of abbreviations

FAP: Health center or prime post

HPT: Human performance technology

PI: Performance improvement

Competing interests

None declared.

Authors' contributions

AF designed the study, analyzed and interpreted the data and wrote a first draft of results. LV provided the Performance Improvement framework and definitions, wrote and edited substantial sections of the paper and contributed to interpretations and conclusions. Both authors read and approved the final manuscript.