User experience analysis of e-TB Manager, a nationwide electronic tuberculosis recording and reporting system in Ukraine.

Ukraine has successfully implemented e-TB Manager nationwide as its mandatory national tuberculosis registry after first introducing it in 2009. Our objective was to perform an end-of-programme evaluation after formal handover of the registry administration to Ukraine's Centre for Disease Control in 2015. We conducted a nationwide, cross-sectional, anonymous, 18-point user experience survey, and stratified the registry's transaction statistics to demonstrate usability. Contrary to initial implementation experience, older users (aged >50 years), often with limited or no computer proficiency prior to using the registry, had significantly better user experience scores for at least six of the 12 measures compared to younger users (aged 18-29 years). Using the registry for >3 years was associated with significantly higher scores for having capacity, adequacy of training received and satisfaction with the registry. Of the 5.9 million transactions over a 4-year period, nine out of 24 oblasts (regions) and Kiev city accounted for 62.5% of all transactions, and corresponded to 59% of Ukraine's tuberculosis burden. There were 437 unique active users in 486 rayons (districts) of Ukraine, demonstrating extensive reach. Our key findings complement the World Health Organization and European Respiratory Society's agenda for action on digital health to help implement the End TB Strategy.

Introduction

The agenda for action of the World Health Organization (WHO) and the European Respiratory Society (ERS) for the End TB Strategy seeks digital health solutions to help advance patient care and improve surveillance and completeness of reporting on treatment outcomes [1]. Digital or electronic health applications to manage multidrug resistant tuberculosis (MDR-TB) can contribute to large-scale implementation of new diagnostics and novel medicines, particularly in resource-constrained countries [2]. One such electronic, web-based system is e-TB Manager, which manages much of the information needed by a country's national TB control programme. Developed by Management Sciences for Health (MSH, Arlington, VA, USA) with funding from the United States Agency for International Development (USAID), e-TB Manager integrates data across all aspects of TB control, including information on presumptive cases, patients, medicines, laboratory testing, diagnosis, treatment and outcome [3]. First developed and implemented in Brazil in 2004, e-TB Manager has been steadily implemented in more than a dozen resource-constrained countries including Ukraine, which is among the 30 high-burden MDR-TB countries worldwide, and the second highest in the European region [4]. After the institutionalisation of e-TB Manager following the end of international and south-to-south technical support in a given country, the source code and technical documentation is handed over to local authorities.

In 2008, as Ukraine confronted its growing TB and MDR-TB burden, it had no electronic system for its vast TB programmes, which were managed by various government agencies, including the state penitentiary system. The quality of the existing paper-based information systems varied. TB control was hampered by weak information, tracking and reporting systems. In response to a request from Ukraine's Ministry of Health, the USAID-funded Strengthening Pharmaceutical Systems programme, implemented by MSH, conducted an initial assessment of customisation needs for e-TB Manager and developed a version suitable for Ukraine. In 2009, e-TB Manager was piloted in six oblasts (regions). A 2010 WHO review of Ukraine's TB programme acknowledged e-TB Manager's beneficial role in supporting management decisions on improving programme performance, including information on case notification and treatment outcomes [5]. In 2010 and 2011, MSH continued to refine e-TB Manager based on user and Ministry of Health feedback, stabilised the platform to ensure continuous access, and trained pilot oblast users in data entry and use. In 2012, in accordance with Ukrainian law on patient confidentiality and information security, the state service of special communication and information protection issued a security certificate for e-TB Manager. With certification complete, the Ministry of Health designated e-TB Manager as the official “national TB registry”. A companion Ministry of Health order specified official adoption, authorisation and requirements for use as well as reports to be produced.

Despite the Euromaidan revolution and socioeconomic crisis, the registry was implemented in all 24 oblasts of Ukraine and the city of Kiev by 2014. In October 2015, following >7 years of international assistance and partnership, the USAID-funded Systems for Improved Access to Pharmaceuticals and Services (SIAPS) Programme, implemented by MSH, handed over the administration of the registry to the Ministry of Health's Ukrainian Centre for Disease Control (UCDC) [6]. The operation and maintenance of the registry is now funded partially by the government of Ukraine and partially by the Global Fund to Fight AIDS, Tuberculosis and Malaria. The objective of this article is to present key findings from our end-of-programme evaluation and share lessons learned from an implementer perspective, sought by the WHO/ERS task force statement, and for donors alike [7-10].

Methods

Our end-of-programme evaluation, part of the institutionally approved work plan and conducted according to the principles of the Declaration of Helsinki [11], drew upon selected usability methods [12]. We retrieved data to demonstrate the usability of the registry through a stratified analysis of geographical and anonymous TB health unit transaction statistics. No patient-related data were analysed. In addition, we retrieved and translated cross-sectional data from the UCDC that measured completeness of reporting by comparing paper-based records with those from the registry. A cross-sectional, anonymous, nationwide survey was conducted to assess the overall user experience of the registry. Our adapted questionnaire had six items on user characteristics, 12 core questions with a 0–7 scale (strongly disagree to strongly agree), and an open text box for comments (see online supplementary material) [13]. The UCDC administered the user experience survey in September and October 2015 and provided the survey's hyperlink to all active users.

Data analysis

Our hypothesis was that years of experience using the registry and years of experience working in the TB programme could influence user experience positively, as reflected in mean scores for the dependent variables (12 core questions). If a user is not knowledgeable about programmatic and clinical management of MDR-TB and its associated recording and reporting procedures necessary for both paper-based systems and electronic applications, the job can be difficult [14, 15]. During the registry's initial implementation, older physicians and health workers resisted using the registry, had limited or no computer proficiency or preferred paper-based systems. During nationwide implementation, the UCDC expected better quality and timeliness of reporting from the rayons (cities and districts of administration) of Ukraine. Therefore, we compared the mean scores of the core questions among user groups based on years of experience using the registry, years working in a TB programme, age and location. We used t-tests to compare the mean scores among two groups of users and ANOVA to compare more than two groups using Tukey's post hoc test [16]. We also investigated three-way interactions and provided effect size values [17]. Statistical analysis was performed using SPSS (version 22.0; SPSS, Chicago, IL, USA). Only completed responses for all required questions were analysed. Anonymous, open-text comments were coded and analysed based on frequency and correlation of paired themes using NVivo 11 (QSR International, Melbourne, Australia) and presented visually [18].

Results

Survey findings

We received 329 responses from 565 active users (58.2% initial response rate), of which 303 included completed responses to all required questions (53.6% usable response rate). Cronbach's α was 0.81 for all 12 questions, and 0.85 for the 10 questions without the two reverse-worded questions, which indicated the questionnaire's high internal consistency. Table 1 provides demographic data of the respondents. Users with >3 years of experience using the registry generally had higher scores than those with <2.5 years of experience, and significance was detected for six out of 12 questions (table 2). Users with <2 years of experience working in the TB programme had relatively lower mean scores for all questions than those with more experience (table 3). For the two reverse-worded questions (Q9 and Q10), where lower scores are better and indicate the level of disagreement, those with 5–10 years of experience had the lowest scores and differed significantly from those with <2 years of experience.

TABLE 1

Demographic characteristics of respondents

Subjects	303
Sex
Female	247 (81.5)
Male	56 (18.5)
Age group years
18–29	32 (10.6)
30–39	96 (31.7)
40–49	87 (28.7)
≥50	88 (29.0)
Location#
National	2 (0.7)
Oblasts	79 (26.7)
Rayons	215 (72.6)
Years working in the TB programme
Median	4
Mean±sd	7.1±6.7
Years using national TB registry
Median	3
Mean±sd	2.7±1.2
Frequently used modules in the national TB registry
Cases	293 (96.7)
Medicines	88 (29.0)
Management	177 (58.4)
Administration	26 (8.6)

Data are presented as n or n (%), unless otherwise stated. TB: tuberculosis. #: n=296.

TABLE 2

Comparison of responses by number of years using the national tuberculosis (TB) registry

	Total	Use of TB registry years		p-value
		<2.5	>3
Subjects n	303	129	174
Q1: I am satisfied with the national TB registry	5.55±1.41	5.32±1.54	5.72±1.28	0.018*
Q2: I have the required capacity to use all features of the national TB registry linked to my responsibilities	5.66±1.35	5.43±1.48	5.84±1.22	0.011*
Q3: I do not need more training on the national TB registry	3.93±2.40	3.94±2.36	3.92±2.44	0.94
Q4: I am happy with the available support and infrastructure for the national TB registry	4.91±1.88	4.75±1.94	5.02±1.84	0.22
Q5: It does not take me long to enter or find information in the national TB registry	5.16±1.88	4.97±2.04	5.30±1.75	0.13
Q6: The national TB registry helps me to improve case management	6.14±1.30	5.92±1.48	6.30±1.13	0.015*
Q7: The training I received on the national TB registry is adequate	4.89±1.99	4.60±2.10	5.11±1.88	0.032*
Q8: The information needed for case management is available in the national TB registry	6.06±1.25	5.81±1.40	6.24±1.10	0.005**
Q9: Generating reports from the paper system is faster than the national TB registry#	2.59±2.51	2.89±2.55	2.37±2.47	0.075
Q10: The national TB registry does not help me identify errors or inaccuracies in patient files#	3.01±2.52	3.10±2.48	2.94±2.56	0.575
Q11: My workplace productivity has improved because of the national TB registry	5.73±1.60	5.52±1.73	5.89±1.49	0.055
Q12: The national TB registry is reliable	5.82±1.41	5.60±1.50	5.99±1.31	0.019*

Data are presented as mean±sd, unless otherwise stated. A scale of 0–7 was used, where 0=strongly disagree and 7=strongly agree. Respondents were shown two ends of the scale for each question and were asked to rate their responses accordingly. #: reverse-worded questions; *: p<0.05, **: p<0.01.

TABLE 3

Comparison of responses by number of years worked in the national tuberculosis (TB) programme

	Total	Work in TB programme years				Post hoc test
		①<2	②3–4	③5–10	④>11
Subjects n	303	78	85	72	68
Q1: I am satisfied with the national TB registry	5.55±1.41	5.28±1.70	5.64±1.38	5.76±1.25	5.51±1.21
Q2: I have the required capacity to use all features of the national TB registry linked to my responsibilities	5.66±1.35	5.10±1.71	5.74±1.32	6.06±1.03	5.79±1.00	②>①* p=0.011;③>①*** p=0.000;④>①** p=0.009
Q3: I do not need more training on the national TB registry	3.93±2.40	3.99±2.32	3.87±2.46	3.94±2.41	3.91±2.47
Q4: I am happy with the available support and infrastructure for the national TB registry	4.91±1.88	4.62±1.95	5.11±1.76	4.94±1.89	4.96±1.95
Q5: It does not take me long to enter or find information in the national TB registry	5.16±1.88	4.87±1.91	5.22±1.94	5.21±1.98	5.37±1.67
Q6: The national TB registry helps me to improve case management	6.14±1.30	5.65±1.77	6.28±1.15	6.35±1.03	6.31±0.95	②>①** p=0.010;③>①** p=0.006;④>①** p=0.012
Q7: The training I received on the national TB registry is adequate	4.89±1.99	4.60±1.93	4.99±2.08	5.32±1.60	4.66±2.24
Q8: The information needed for case management is available in the national TB registry	6.06±1.25	5.79±1.45	6.21±1.08	6.22±0.96	6.00±1.43
Q9: Generating reports from the paper system is faster than the national TB registry#	2.59±2.51	3.29±2.64	2.54±2.54	2.13±2.34	2.34±2.38	③<①* p=0.023
Q10: The national TB registry does not help me identify errors or inaccuracies in patient files#	3.01±2.52	3.59±2.54	3.14±2.65	2.51±2.36	2.69±2.40	③<①* p=0.044
Q11: My workplace productivity has improved because of the national TB registry	5.73±1.60	5.41±1.81	5.67±1.76	6.07±1.24	5.81±1.43
Q12: The national TB registry is reliable	5.82±1.41	5.54±1.51	6.06±1.15	5.90±1.46	5.76±1.48

Data are presented as mean±sd, unless otherwise stated. A scale of 0–7 was used, where 0=strongly disagree and 7=strongly agree. Respondents were shown two ends of the scale for each question and were asked to rate their responses accordingly. #: reverse-worded questions. *: p<0.05, **: p<0.01; ***: p<0.001 (Tukey's honest significant difference).

Users aged >50 years had higher scores for at least eight questions and differed significantly from users aged 18–29 years for six out of the 12 questions (table 4). Users aged 30–39 years differed from their peers in their opinion that the registry does not help identify errors or inaccuracies in patient files (Q10), and the difference from users aged 18–29 years was significant. The comparison of mean scores between users at the oblast and rayon levels is presented in the online supplementary material. Significant findings for the three-way ANOVA examining the effects of years using the registry and years working in a TB programme, each with age and location, are presented in table 5. While there were main effects for years of experience using the registry with the selected variables and predominantly significant two-way interactions, the effect size, as expressed in η2, was small. There was a medium effect size for age with years working in the TB programme concerning the belief that generating reports using a paper system is faster than the registry. Figures 1–3 and table 6 present the qualitative analysis of the relationship between major themes based on feedback from 74 out of 303 respondents.

TABLE 4

Comparison of responses by age categories

	Total	Age categories years				Post hoc test
		①18–29	②30–39	③40–49	④>50
Subjects n	303	32	96	87	88
Q1: I am satisfied with the national TB registry	5.55±1.41	4.88±2.21	5.52±1.35	5.57±1.19	5.80±1.25	④>① p=0.009*
Q2: I have the required capacity to use all features of the national TB registry linked to my responsibilities	5.66±1.35	5.69±1.59	5.59±1.30	5.71±1.21	5.68±1.45
Q3: I do not need more training on the national TB registry	3.93±2.40	3.56±2.46	3.68±2.40	4.07±2.43	4.19±2.35
Q4: I am happy with the available support and infrastructure for the national TB registry	4.91±1.88	4.13±2.18	4.82±1.81	5.00±1.77	5.19±1.91	④>① p=0.031*
Q5: It does not take me long to enter or find information in the national TB registry	5.16±1.88	4.44±2.36	5.15±1.89	5.16±1.75	5.44±1.76	④>① p=0.048*
Q6: The national TB registry helps me to improve case management	6.14±1.30	5.53±1.79	6.08±1.19	6.17±1.29	6.40±1.16	④>① p=0.007**
Q7: The training I received on the national TB registry is adequate	4.89±1.99	4.78±2.13	4.60±2.09	5.08±1.93	5.07±1.88
Q8: The information needed for case management is available in the national TB registry	6.06±1.25	5.53±1.81	6.05±1.17	6.06±1.26	6.26±1.03	④>① p=0.024*
Q9: Generating reports from the paper system is faster than the national TB registry#	2.59±2.51	1.78±2.39	2.91±2.64	2.31±2.23	2.82±2.62
Q10: The national TB registry does not help me identify errors or inaccuracies in patient files#	3.01±2.52	2.13±2.39	3.56±2.62	3.08±2.43	2.65±2.43	②>① p=0.026*
Q11: My workplace productivity has improved because of the national TB registry	5.73±1.60	5.13±2.24	5.56±1.72	5.78±1.47	6.08±1.21	④>① p=0.020*
Q12: The national TB registry is reliable	5.82±1.41	5.50±1.93	5.68±1.49	5.85±1.22	6.07±1.23

Data are presented as mean±sd, unless otherwise stated. A scale of 0–7 was used, where 0=strongly disagree and 7=strongly agree. Respondents were shown two ends of the scale for each question and were asked to rate their responses accordingly. #: reverse-worded questions. *: p<0.05, **: p<0.01 (Tukey's honest significant difference).

TABLE 5

Significant findings of three-way ANOVA

	Years using registry*age*location	Years working in TB programme*age*location
Q2: I have the required capacity to use all features of the national TB registry linked to my responsibilities	Main effect for years using registry:F (1, 279)=4.32, p<0.05, η2=.01No two-way interaction was significant	Main effect for years working in TB programme: F (3, 274)=5.07, p<0.01, η2=.05No two-way interaction was significant
Q3: I do not need more training on the national TB registry	Two-way interaction for years using registry*location:F (1, 279)=4.98, p<0.05, η2=.02
Q4: I am happy with the available support and infrastructure for the national TB registry	Main effect for age: F (3, 279)=2.74, p=0.04, η2=.03Two-way interaction for age*location:F (3, 279)=2.75, p<0.05, η2=.03
Q6: The national TB registry helps me to improve case management	Main effect for age: F (3, 279)=3.39, p<0.05, η2=.03Two-way interaction for age*location:F (3, 279)=3.87, p<0.05, η2=.04	Main effect for years working in TB programme: F (3, 265)=4.78, p<0.01 η2=.05No two-way interaction was significant
Q8: The information needed for case management is available in the national TB registry	Two-way interaction for age*location:F (3, 279)=3.76, p<0.05, η2=.04
Q9: Generating reports from the paper system is faster than the national TB registry#	Main effect for location: F (2, 279)=4.85, p<0.01, η2=.03Three-way interaction for years using registry*age*location: F (3, 279)=4.35, p<0.01. η2=.04	Main effect for location: F (2, 265)=3.58, p=0.02, η2=.02Two-way interaction for age*years in TB programme:F (8, 265)=2.74, p<0.01, η2=.08
Q10: The national TB registry does not help me identify errors or inaccuracies in patient files#	Main effect for age: F (3, 279)=2.86, p<0.05, η2=.03Main effect for location: F (2, 279)=5.50, p<0.01, η2=.04Two-way interaction for age*location:F (3, 279)=2.97, p<0.05, η2=.03	Main effect for age: F (3, 265)=3.03, p=0.03, η2=.03Main effect for location: F (2, 265)=4.48, p<0.01, η2=.03No two-way interaction was significant
Q11: My workplace productivity has improved because of the national TB registry	Main effect for age: F (3, 279)=3.61, p<0.01, η2=.03Two-way interaction for age*location:F (3, 279)=4.72, p<0.01, η2=.04
Q12: The national TB registry is reliable	Two-way interaction for age*location:F (3, 279)=3.28, p<0.05, η2=.03

TB: tuberculosis. #: reverse-worded questions.

FIGURE 1

Most frequent words and phrases based on user comments about the registry. Word cloud representing the most frequent words based on 74 user comments about the registry. The font size represents how frequently the terms occur. Only stemmed words were utilised, with a minimum length of three words; the word “registry” was not included in this analysis.

FIGURE 2

Major themes from the user comments. Comments from 74 users were coded into 11 major themes. #: server or registry platform; ¶: paper-based reports and registry data fields; +: suggestions on registry structure.

FIGURE 3

Dendrogram: themes clustered by word similarity. Themes clustered by word similarity: the dendogram is the result of a cluster analysis of user comments that clustered selected themes together if they have many words in common. Similar items are clustered together on the same branch and different items are positioned further apart. #: server or registry platform; ¶: suggestions on registry structure; +: paper-based reports and registry data fields.

TABLE 6

Correlation of key themes

Theme A	Theme B	Pearson's correlation coefficient
Patient aspects	Data quality	0.57
Physician resistance	Computer access needed	0.53
Physician resistance	Data quality	0.52
Retraining needed	Physician resistance	0.50
System issues	Retraining needed	0.48
System issues	Physician resistance	0.45
Retraining needed	Data quality	0.43
Physician resistance	Patient aspects	0.43
Data quality	Benefits	0.43
System issues	Data quality	0.42
Physician resistance	Expand registry access	0.41
Retraining needed	Expand registry access	0.39
Reporting issues	Data quality	0.37
Data quality	Data entry issues	0.37

There were 55 possible correlations of various themes that ranged from 0.08 to 0.57. This table represents the first 14 paired themes in rank order of correlation.

Stratified usability analysis

Figure 4 presents the distribution of active users and average transactions per active user for each oblast stratified by TB burden. For example, the Lviv oblast with 91 active users had an average transaction of 3000 per active user, compared to the Mykolaiv oblast with half as many active users with an average transaction of 10 000 per active user. Nine out of 24 oblasts and Kiev city accounted for 62.5% of user transactions and 59% of Ukraine's TB burden. Online supplementary table S4 lists anonymous usability statistics among active registry users since inception for all 24 oblasts and Kiev (5.90 million cumulative transactions). As of June 2016, there were nearly 2000 registered users across 722 registered TB units nationwide. However, only 1319 users across 647 active TB units are considered active, because they logged at least one transaction in the previous 12 months. The highest average transactions per TB unit occurred in the Donetsk oblast (27 110), followed by Kiev city (19 037). Online supplementary table S4 highlights the extent of the registry's nationwide user coverage beyond oblast and city administration, particularly in rayons, towns and other cities. Figure 5 illustrates cumulative usability statistics for selected data fields from the registry's case module among the top 10 TB units by transaction volume. The UCDC-monitored, -analysed and -reported data on correspondence of paper-based records with the registry is provided in the online supplementary material.

FIGURE 4

Usability statistics among active users of the national tuberculosis (TB) registry by TB burden. Each oblast (region) lists the number of active registry users in June 2016. Numbers in parentheses indicate average transactions per user in thousands (k). For example, the Kherson oblast with a medium TB burden has 50 active users with an average of 6000 transactions per user. Registry transactions are cumulative, from 2011 to June 2016 (online supplementary material). Nine out of 24 oblasts and Kiev accounted for 62.5% of cumulative transactions and 59% of the TB burden.

FIGURE 5

Usability statistics among top 10 tuberculosis (TB) units by transaction type for key data fields. This figure presents anonymous cumulative transaction statistics for selected data fields from the registry's case module from January 2012 to May 2016. The top 10 out of 647 active TB health units, accounting for nearly a third of the total cumulative transaction volume (5.9 million) are presented (online supplementary material). DST: drug susceptibility testing; LPA: line probe assay.

Discussion

Our study sheds light on several factors associated with overall user experience and usability of the registry. Users gave high ratings to the registry's ability to help improve case management. Upon entering the registry, a dashboard presents real-time data for the patients being managed at that facility in areas such as case notification, treatment progress, medicines adherence, status of medicines and diagnostic supplies and tags to flag data quality issues. From an oblast administrator perspective, the chief TB doctor responsible for validating treatment protocols can easily review individual cases and laboratory results, validate the treatment regimen, remotely monitor adherence to clinical guidelines and take corrective action by using alerts embedded in the registry. Users strongly agreed that the information needed for case management is available in the registry. This is not surprising, given that the number of cases entered in the registry substantially increased from 120 472 in 2014 to 227 657 in 2016. A WHO assessment found that the proportion of missing information in the registry did not exceed 10% [19]. Complemented by the UCDC's analysis of data correspondence of paper-based reports with that of the registry (online supplementary material) and the usability statistics, the survey findings confirm that changes in the human element and behaviour change over time were important factors in users embracing the registry. Users with >3 years of experience with the registry at the rayon level, or aged >50 years had significantly higher mean scores for agreeing that information is available in the registry. By contrast, oblast-level users had significantly lower scores, possibly because these users are generally supervisors or administrators responsible for aggregating data, and are more likely to detect data quality issues such as for accuracy or completeness.

Users with >3 years of registry experience had significantly higher scores than users with less experience for satisfaction, capacity, adequacy of training received and perceived reliability of the registry; this finding is associated with longer use of electronic applications [20]. There was also a main effect on having capacity for number of years using the registry and number of years working in a TB programme, suggesting that with more use and experience, regardless of age, the user eventually gets comfortable with the registry and understands UCDC recording and reporting expectations. Contrary to the finding of another study [21], we found that older users generally had a positive user experience according to many of the variables and differed significantly from users aged 18–29 years, while another study reported no effect for age [22]. The WHO has promoted electronic TB recording and reporting to improve surveillance and support meaningful decision making based on access to accurate, timely and useful information [23]. Consequently, there was generally a high rating for improved workplace productivity due to the registry, particularly among older users. Management and clinical decisions can be made easily, reducing time to decision, as was briefly described in one oblast of Ukraine [24]. One chief TB doctor reported that the time taken to initiate antiretroviral therapy among TB patients dropped from 104 days to 48 days due to the registry.

With system issues (e.g. slow server or too many registry updates) being the dominant theme among user comments (figures 1–3), younger users probably have unmet expectations, including the need for more computers to minimise sharing. Compared to users aged 18–29 years, older users probably have lower expectations and have experienced tougher working conditions [25]. The USAID SIAPS programme and the Global Fund to Fight AIDS, Tuberculosis and Malaria initially supported the purchase of 534 computers, but our survey found a moderate correlation between physician resistance and computer access (figures 1–3 and online supplementary material), which is consistent with previous studies [26]. Over the years, the UCDC has kept pace with necessary programmatic changes based on WHO guidelines and updated reporting requirements and upgraded the registry based on user feedback. From 2012 to 2016, 137 system updates were made for continuous quality improvement of the registry. The 10 most recent registry updates with pictorial instructions is a good example of e-learning methodologies promoted by the WHO and ERS [27]. A dedicated staff at the UCDC's national helpdesk fields on average 150 phone calls and emails a week, and after updates are released, this can jump to >1200 calls and emails each week. Yet, regardless of user characteristics, the overall mean scores on support and infrastructure were low. Additionally, there was a high level of user disagreement, as expressed in low mean scores on not needing more training and the moderate correlation of retraining needs with physician resistance, system issues and data quality (figures 1–3 and online supplementary table S3). Despite a relatively high score on having capacity, users expect annual training that can also serve to share experiences and improve data quality and reporting. However, training budgets in resource-constrained public-sector settings are limited. Increasing usage presents new challenges in the provision of user support. On-the-job mentoring, refresher training and supportive supervision contribute to a better user experience.

Our study findings have implications for other implementers interested in applying digital health applications in resource-constrained settings. In our experience, building sustained user (individual) and institutional capacity took >5 years, particularly when dealing with poor computer literacy, resistance to change and infrastructure challenges, among others. More research must be undertaken to investigate the root cause of persistent physician resistance. Future studies must compare differences between nurses, laboratory technicians, statisticians/data analysts, pharmacists and administrators. Such analysis could contribute to the development of an implementation model based on workflow design and job responsibilities for digital health application in other resource-constrained settings.

As with any observational study, our findings must be interpreted with caution (online supplementary material). Some or many of the findings presented here may have been addressed by the UCDC in the months following the cross-sectional survey. Although 66% of nearly 2000 registered users (June 2016 data) were considered active based on at least one transaction in the previous 12 months, a limitation of the usability statistics is that we do not account for passive users. We are unable to determine the registered passive users who could be senior ministry personnel who occasionally view the data, supervisors who only generate reports, university staff or researchers, who do not perform transactions, but were nevertheless granted access to the registry.

This is the first elaborate country-level study on user experience and usability of e-TB Manager in resource-constrained settings, and, to our knowledge, the first from Ukraine in the English language. Our end-of-programme findings are significant, given that Ukraine's global ranking on overall government usage of information communication technologies (ICTs) declined from 56 in 2009 to 124 in 2015 [28]. Ukraine's global ranking for political and regulatory environment for ICTs also declined from 95 in 2009 to 122 in 2015 (the same years as the nationwide scale-up of the registry). Notably, at the time of the survey, there were 565 active users, and this increased to 1319 active users over the following 9 months, illustrating the expansion of the registry to more TB units and rayons of Ukraine and increasing usage. The UCDC, now part of the public health centre, provided strong leadership and commitment to implementing the national TB registry despite political unrest, military conflict and socioeconomic challenges while confronting the MDR-TB burden [29-31]. In conclusion, Ukraine's experience and resilience in implementing its national TB registry using the e-TB Manager platform is testament to the WHO/ERS's digital health agenda to help implement the End TB Strategy and a model for other resource-constrained countries.

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

00002-2017_Online_Supplement

N. Konduri 00002-2017_Konduri

N. Nizova 00002-2017_Nizova

K. Sawyer 00002-2017_Sawyer