Analyzing repositories of OER using web analytics and accessibility tools.

Open Educational Resources (OER) provide learning opportunities for all. Usually, OER and links to OER are curated in Repositories of OER (ROER) for open access and use by anyone, including people with disabilities, at any place at any time. This study analyzes the reputation/ authoritativeness, usage, and accessibility of thirteen popular ROER for teaching and learning using three Web Analytics and five Web Accessibility tools. A high difference among the ROER was observed in almost every metric. Millions of users visit some of these ROER every month and on average stay 2-26 min per visit and view 1.1-8.5 pages per visit. Although in many ROER most of their visitors come from the country where the ROER hosting institute operates, other ROER (such as DOER, MIT OCW, and OpenLearn) have managed to attract visitors from all over the world. In some ROER, their visitors come directly to their website while in a few other ROER visitors are coming after visiting a search engine. Although most ROER are accessible by users with disabilities, the Web Accessibility tools revealed several errors in few ROER. In most ROER, less than one third of the traffic is coming from mobile devices although almost everyone has a mobile phone nowadays. Finally, the study makes suggestions to ROER administrators such as interconnecting their ROER, collaborating, exchanging good practices (such as Commons and MIT OCW), improving their website accessibility and mobile-optimized design, as well as promoting their ROER to libraries, educational institutes, and organizations.

Introduction

The worldwide educational disruption due to the COVID-19 pandemic forced educational institutes to switch from face-to-face or blended teaching and learning to fully online teaching and learning. More than 1.2 billion students were prevented from going to school during 2020–2021 [32]. However, most educational systems, educational institutes, educators, and learners were not prepared for such an abrupt and unavoidable transition to emergency distance teaching and learning. More specifically, teachers and students lacked educational resources since commercial textbooks are usually not available online through either the publishers or the libraries’ reservation systems. So, there was a need for openly available digital educational materials that could be freely used online by both teachers and students. Open Educational Resources (OER) could satisfy this need.

OER are educational materials under open licenses that can be openly accessed, used, modified, and shared by anyone [46]. According to Creative Commons (CC), OER are “teaching, learning, and research materials that are either (a) in the public domain or (b) licensed in a manner that provides everyone with free and perpetual permission to engage in the 5R activities– retaining, remixing, revising, reusing and redistributing the resources." https://hewlett.org/strategy/open-education/. More specifically, users are free to: (1) Retain the resource (e.g., make, archive, and "own" copies; (2) Reuse it; (3) Revise it (adapt, modify, transform, translate or alter; (4) Remix it (e.g., combine, merge or integrate with other content to create a new content; and (5) Redistribute it (share the original or revised or remixed content) [49] as well as to find it, access it, evaluate it, or even abandon it [17].

OER can reduce economic barriers and inequalities among students since all students can have free access to open textbooks and save money [11, 21, 22, 48]. Furthermore, OER could enable international knowledge exchange, and collaboration. Authors and teachers from various countries can collaborate in authoring OER, developing curriculum, syllabus and lesson plans based on OER, and teaching using OER. They can apply localization to the OER or in other words adapting the OER to their local context (e.g., country, language, culture, educational standards) and adjusting (e.g., translating) it to the context of another educational system.

Furthermore, OER can increase students’ independence, collaboration, interest, and satisfaction (e.g., [10, 13, 15, 38, 39, 48]. OER not only reduce the cost of textbooks (e.g., [9, 22, 23, 27] but also improve student learning (e.g., [12, 20, 25]. For example, Weller et al. [48] found that over 60% of students and teachers agreed or strongly agreed that OER increased students’ satisfaction, interest in the subjects taught, and enthusiasm for future study as well as students’ experimentation with new ways of learning.

However, OER have not spread widely across the world due to several obstacles. One main obstacle is the difficulty to find quality and suitable OER for a specific educational objective, educational level, students’ profile, teaching method, and other prerequisites (e.g., [4, 5, 7, 8, 13, 14, 24, 26, 28, 29, 33, 34, 41].

OER are stored, organized, and curated for later retrieval and use in Repositories of OER (ROER). During the last fifteen years there has been a continuous increase in the number of ROER from 600 (2006) to 1900 (2011) to 3200 (2016) to 5600 (2021) (e.g., [16, 37]. Most of these ROER contain journal articles (there are 3,977 such ROER), theses and dissertations (there are 3284 such ROER), books, chapters and sections (there are 2166 such ROER), conference and workshop papers (there are 1966 such ROER), reports and working papers (there are 1883 such ROER), bibliographic references (there are 868 such ROER), learning objects (there are 785 such ROER), and more. Furthermore, many ROER contain only metadata with hyperlinks to OER or even to other ROER. This study will use the term ROER to mean any repository that contains OER, or metadata with hyperlinks to OER or to other ROER. So, a teacher or a student looking for quality OER to satisfy a specific educational need has to explore many different ROER. However, each ROER has a different structure and describes differently its curated OER from other ROER [33, 34]. Despite the existence of such a high variety of ROER, little is known about their characteristics and their use by visitors such as educators, authors, researchers, librarians, learners, administrators, managers, policy makers, and others. More specifically, there are not many studies that analyze the characteristics, especially with respect to accessibility issues, and usage of ROER for teaching and learning. Therefore, this study attempts to shed light on some characteristics, including accessibility issues, and usage of popular ROER for teaching and learning. More specifically it will try to answer the following research questions (RQ):

What is the usage of popular ROER for teaching and learning?

What is the reputation of popular ROER for teaching and learning?

What is the accessibility of popular ROER for teaching and learning?

Previous studies

Several previous studies have investigated various characteristics of ROER. Atenas and Havemann [6] evaluated ROER across ten quality indicators: Authorship, CC Licences, Featured resources, Keywords, Metadata, Multilingual support, Peer review, Social Media support, Source Code or Original Files, and User evaluation tools. They found that most ROER are located in Europe and North America, are managed by a single institute and rank low in all quality indicators except Keywords. Santos-Hermosa et al. [40] examined ROER in higher education and found that most ROER are multidisciplinary, are based in Europe, use DCpace as a platform, Dublin Core as metadata standard, and Creative Commons as licensing. Navarrete and Luján-Mora [30] compared the accessibility of their own OERfAll website to OER Commons using WAVE. Perifanou and Economides [35] investigated the quality, popularity, and usage of well-known ROER using manual inspection and the following five Web Analytics tools: Google MobileFriendly, Google PageSpeed Insights, OpenLink Profiler, SimilarWeb, and WAVE. They found that the majority of these ROER contain OER and links pointing to OER of multiple types, multiple languages, multiple disciplines (subjects), and multiple educational levels. Also, nearly all of them offer mobile friendly and accessible design, some kind of OER quality review, OER-searching, and members’ collaboration services. Most of them have a large number of registered members and followers on social media. However, the majority offers slow speed and the information they give about their OER is frequently inaccurate. In the same way, Perifanou and Economides [36] analyzed the characteristics, popularity, and visitors’ engagement of twenty major repositories of open textbooks for higher education using manual inspection and the same Web Analytics tools. They found that most repositories curate open textbooks with CC licenses in a variety of disciplines. Most repositories were responsive and accessible. Half of these repositories enable reviews for their open textbooks by users or experts. Typically, most visits to a repository come from users in the country where this repository is established. In addition, most repositories received the most visits after visitors performed a search with a search engine. However, the majority of these repositories did not reach a good bounce rate.

Open education aims at ensuring that everyone has open access to education. Openness in education encompasses the concepts of non-discrimination, inclusiveness, and accessibility [18]. Thus, it is important that ROER are accessible by all, including people with disabilities. Usually, people who have a disability that affects their vision, hearing, motor or cognitive functions use assistive technologies (such as screen readers, alternative keyboards and/or trackpads, screen magnifiers) to navigate and interact with websites. Therefore, the websites should be designed and developed in a way that enables these assistive technologies’ tools to function properly [50]. Performing Web Accessibility tests is a way to uncover design errors on a website that prevents people who rely on assistive technologies to access and interact with the website. However, the current state of ROER with regards to accessibility has not been investigated thoroughly. In a previous study, Perifanou and Economides [35] used WAVE to analyze the accessibility of well-known ROER. The current study extends that study and uses five Web Accessibility tools [1, 3, 19, 43, 47], plus three different Web Analytics tools [2, 42, 44] to explore thirteen ROER. It aims at describing the reputation / authoritativeness, usage, and accessibility of popular ROER for teaching and learning. To the best of our knowledge, no other study investigated the accessibility of many ROER using a variety of Web Accessibility tools. In addition, three different Web Analytics tools were used to further investigate these ROER. The following sections describe the methodology, results, and conclusions of this investigation.

Methodology

This study was conducted from January to May 2021. It followed a six-stages’ methodology (Fig. 1).

Fig. 1

Stages of the methodology

Many universities’ libraries manage websites with useful resources for teaching and learning. These websites also contain suggestions to educators and students about useful ROER that curate quality OER for teaching and learning. At the first stage of the methodology, the authors looked at over one hundred (100) such websites and identified thirteen popular ROER that were the most frequently suggested ROER by the universities’ librarians. The final list of the selected ROER is the following:

Commons: https://lor.instructure.com/

Curriki: https://library.curriki.org/

DOER: http://doer.col.org/

KlasCement: https://www.klascement.net/?hl=en

MERLOT: https://www.merlot.org/merlot/

MIT OCW (OpenCourseware): https://ocw.mit.edu/index.htm

MOM: https://oer.deepwebaccess.com/oer/desktop/en/search.html

OASIS: https://oasis.geneseo.edu/

OER Commons: https://www.oercommons.org

OER World Map: https://oerworldmap.org/

OpenLearn: https://www.open.edu/openlearn/

OpenStax: https://openstax.org/

Open Textbook Library: https://open.umn.edu/opentextbooks/

During the second stage, variables for evaluating a ROER website were defined. The Reputation & Authoritativeness of a ROER is defined to be the extent to which educational institutes as well as educational website developers and managers, educators, authors, librarians, researchers, and others believe that this ROER is an authoritative and credible ROER. Also, the Usage of a ROER is defined to be the extent that users (such as educators, teachers, librarians, authors, students) use this ROER.

At the third stage, three Web Analytics tools, Alexa [2], Semrush [42], and SpyFu [44] were employed in order to analyze the Reputation/ Authoritativeness and Usage of these ROER. Alexa [2] by Amazon is an online ranking tool of websites. Semrush [42] is an all-in-one tool suite for improving companies’ online visibility and marketing strategies. It helps companies run search engine optimization (SEO), pay-per-click (PPC), social media, and content marketing campaigns. SpyFu [44] is used by digital marketers to improve their online advertising strategy. It combines SEO, PPC, backlink outreach, and keyword research.

Furthermore, five Web Accessibility tools, AChecker (WCAG 2 AA) [1], ANDI (Accessible Name & Description Inspector) Accessibility Testing Tool [3], EqualWeb A11y Checker [19], Siteimprove Accessibility Checker [43], and Web Accessibility [47] were used in order to automatically assess the accessibility of the first page of each ROER. Note that for a thorough accessibility test and repair of a website, more than automated tools are required including semi-automated checks as well as manual (user and expert-based) testing. However, the aim of this study is to only investigate if these ROER adhere to basic accessibility guidelines.

AChecker (WCAG 2 AA) [1] is an automated accessibility checker for evaluating the accessibility of HTML pages. It helps to ensure that websites can be accessed by all individuals, including those with disabilities, using assistive technologies to navigate the Internet. ANDI (Accessible Name & Description Inspector) Accessibility Testing Tool (2021) provides automated detection of accessibility issues, tests for 508 compliance and suggests the vocalization for interactive elements and other practices for accessibility improvement. EqualWeb A11y Checker [19] analyzes the website accessibility and makes recommendations on how to fix accessibility problems. It is recognized by the world-wide-web Consortium (W3C) to be fully compliant with their Web Content Accessibility Guidelines [50] WCAG 2.1 and Sect. 508 guidelines. Siteimprove Accessibility Checker [43] checks the web page’s accessibility based on the latest ACT (Accessibility Conformance Testing) rules and makes suggestions for overcoming accessibility problems. Web Accessibility [47] allows to easily check and monitor if a website meets the WCAG 2.1 standards. It runs many automated tests and identifies accessibility violations.

Then at the fourth stage, the following five metrics (provided by the Web Analytics tools Semrush, Alexa, and SpyFu) were used to measure the Reputation & Authoritativeness of the ROER: (1) Authority Score; (2) Referring Domains; (3) Backlinks; (4) Linking Sites; and (5) Ranking (Table 1). It is expected that a highly reputable and authoritative ROER will have a high authority score as well as many websites and links pointing to it. Also, the following ten metrics (provided by the Web Analytics tools Semrush and Alexa) were used to measure the Usage of the ROER: (1) Monthly Visits; (2) Traffic Distribution by Country; (3) Percentage Visits by Direct, Referral or Search; (4) Number of Unique Visitors; (5) Mobile; (6) Number of Pages per Visit; (7) Daily Page views per visitor; (8) Average Visit Duration; (9) Daily time on site; and (10) Bounce Rate (percentage of visitors that leave the website after viewing just one page) (Table 1). It is expected that a highly used ROER will attract many visitors per month who will visit many pages and stay for long time on it. Finally, the accessibility of the first page of each ROER was measured by the following fourteen metrics (provided by the five Web Accessibility tools): (1) Number of Issues: Specific Issues; (2) Focusable Elements; (3) Accessibility Alerts; (4) General Errors; (5) Contrast Errors; (6) Notices; (7) Warnings; (8) Aria Attribute; (9) Role attribute; (10) Known Problems; (11) Likely Problems; (12) Potential Problems; (13) Health Score; and (14) Violations. It is expected that an accessible ROER will have none or very few accessibility problems.

Table 1

Metrics to measure ROER reputation/authoritativeness, usage, and accessibility

ROER evaluation variables	ROER metrics provided by Web Analytics and Accessibility Tools
Reputation and authoritativeness	Authority Score (Semrush)
	Referring Domains (Semrush)
	Backlinks (Semrush)
	Linking Sites into (Alexa)
	Ranking (SpyFu)
Usage	Monthly Visits (Semrush)
	Traffic Distribution by Country (Semrush)
	Percentage Visits by Direct, Referral or Search (Semrush)
	Number of Unique Visitors (Semrush)
	Mobile access (Semrush)
	Number of Pages per Visit (Semrush)
	Daily Page views per visitor (Alexa)
	Average Visit Duration (Semrush)
	Daily time on site (Alexa)
	Bounce Rate (Semrush)
	Bounce Rate (Alexa)
Accessibility	Number of Issues and Specific Issues (Siteimprove Accessibility Checker)
	Focusable Elements and Accessibility Alerts (ANDI Accessibility Testing Tool)
	General Errors; Contrast Errors; Notices; Warnings; Aria Attribute; Role attribute (EqualWeb A11y Checker)
	Known Problems; Likely Problems; Potential Problems (AChecker (WCAG 2 AA)
	Health Score and Violations (Web Accessibility)

The following definitions describe the metrics [2, 42, 44]:

Average visit duration is the average amount of time that a visitor spends viewing ROER pages during a single visit to the ROER website;

Authority score is a composite metric used to measure the overall quality and influence of a ROER’s domain on search engine optimization (SEO). The score is based on the number of domain backlinks, number of referring domains, number of referring IPs, correlation between domain score and trust score, link follow vs no follow, organic, and number of users. The higher this number is the better;

Backlinks are incoming links to a ROER website from an external page. An external website may have many hyperlinks to the ROER website. Backlinks are one of the two most important metrics that Google uses to rank websites. However, there are many types of backlinks of different quality;

Bounce rate is the percentage of visitors who visit one page of a ROER website and then leave the ROER website without visiting other pages;

Daily page views per visitor is the average number of ROER page views by a visitor in a day;

Daily time on site is the average amount of time that a visitor spends on a ROER website in a day;

Direct traffic is the traffic that comes to a ROER website via URLs (Universal Resource Locators) entered in a browser’s search bar, saved bookmarks or links from outside a browser (such as PDFs or Microsoft Word documents) over the past six months;

Linking sites into or referring domains are the number of websites or domains that point to a ROER website;

Mobile refers to the percentage of traffic coming from mobile devices;

Monthly visits refer to the average number of visitors to a ROER website per month;

Number of pages per visit is the average number of ROER pages that a visitor views during a single visit to the ROER website;

Number of unique visitors to a ROER website in a month. Note that a unique visitor may come multiples times to the ROER website during a month but he/she is counted once;

Ranking of a ROER website in comparison with other websites. The lower this number, the better;

Referral traffic is the traffic that comes to a ROER website from a web source outside of search engine and social media (e.g., when a person clicks on a hyperlink from another website that leads to the ROER website) over the past six months;

Search traffic is the traffic that comes to a ROER website directly from a search engine over the past six months;

Traffic distribution by country is the percentage of traffic that comes to a ROER website from various country (Table 2 presents the top three countries sending traffic to each ROER).

Table 2

Measuring ROER usage using Semrush

Repositories of OER (ROER)	Monthly visits	Traffic distribution	Percentage visits: direct referral search (%)	Unique visitors	Pages/visit	Avg visit duration	Bounce rate (%)
Commons	348.6 M	US: 83%	63	22.1 M	8.45	26:00	18
		Philippines: 3%	33
		Mexico: 1%	3
Curriki	13.9 K	US: 96%	73	12 K	1.09	05:59	91
		Indonesia: 2%	8
		Egypt: 1%	0
DOER	30.3 K	India: 28%	28	22 K	1.71	02:28	65
		Malaysia: 18%	3
		Canada: 18%	66
KlasCement	654.1 K	Belgium: 83%	35	347.7 K	4.99	08:23	45
		Netherlands: 15%	6
		Suriname: 3%	59
MERLOT	142.5 K	US: 77%	35	104.8 K	3.11	09:43	62
		Philippines:5%	21
		Canada: 3%	44
MIT OCW	47.8 M	US: 47%	46	20.3 M	5.54	15:44	46
		India: 6%	9
		UK: 5%	41
MOM	5 K	US: 73%	19	4.7 K	2.37	04:08	32
		S. Korea: 14%	60
		Canada: 10%	21
OASIS	625.2 K	US: 87%	52	209.1 K	6.15	16:50	42
		India: 3%	15
		Philippines: 2%	32
OER commons	210 K	US: 78%	42	119 K	3.14	09:38	61
		India: 6%	24
		Philippines: 3%	32
OER world map	5.1 K	Germany: 72%	36	3.2 K	2.29	07:01	79
		India: 16%	0
		UK: 10%	46
OpenLearn	3 M	UK: 39%	35	2 M	4.53	12:51	61
		US: 16%	16
		India: 10%	46
OpenStax	2.4 M	US: 76%	50	1.5 M	1.79	07:32	72
		India: 5%	18
		Canada: 4%	31
Open textbook library	20.2 M	US: 82%	55	7.1 M	5.78	18:31	42
		India: 3%	11
		S. Korea: 2%	33

Data analysis and results

During the fifth stage, these thirteen ROER were analyzed using the three Web Analytics tools and the five Web Accessibility tools. The findings revealed large differences between these ROER across all metrics (Tables 2, 3, 4). The last stage is the reporting of the findings.

Table 3

Measuring ROER reputation and authoritativeness using Semrush

Repositories of OER (ROER)	Authority score	Ref. domains	Backlinks	Mobile access (%)
Commons	80	1.13 K	49.1 K	9
Curriki	59	491	69.7 K	38
DOER	60	97	1.2 K	39
KlasCement	61	3.3 K	11 M	34
MERLOT	69	13.5 K	5.1 M	13
MIT OCW	85	50.27 K	9 M	27
MOM	38	330	55 K	5
OASIS	61	467	8.3 K	31
OER commons	65	10.09 K	5.8 M	21
OER world map	38	589	13.4 K	0
OpenLearn	68	191	1.7 K	50
OpenStax	65	9.4 K	1.1 M	28
Open textbook library	79	6.89 K	461.6 K	24

Table 4

Measuring ROER Reputation and Usage using SpyFu and Alexa

Repositories of OER (ROER)	1/3/2019 ranking	1/3/2020 ranking	1/3/2021 ranking	Daily page views per visitor	Daily time on site	Bounce rate (%)	Linking sites into
Commons	> 51	46.7	28.7	10.6	12:17	16	1034
Curriki	> 51	> 51	29.1	1.8	1:06	75	926
DOER	49.6	44.8	37.1	2.2	3:16	60	711
KlasCement	> 51	49.2	31.0	4.3	3:31	37	258
MERLOT	> 51	38.9	26.4	2.7	2:59	53	1401
MIT OCW	23.3	14.9	15.1	4.12	5:41	42	65,394
OASIS	> 51	> 51	> 51	1.4	1:45	78	971
OER commons	30.7	17.7	18.1	3.8	3:31	43	759
OER world map	> 51	> 51	31.9	2	4:23	63	7
OpenDOAR	> 51	> 51	23.8	3.4	2:11	35	1612
OpenLearn	43.1	21.7	17.2	1.8	2:22	72	1683
OpenStax	43.1	37.3	19.7	3.68	4:05	53	100
Open textbook library	> 51	> 51	5.33	3.9	3:45	52	27,445

Usage of the 13 ROER

Regarding ROER Usage, Semrush showed (Table 2) that Commons, MIT OCW, and Open Textbook Library received the most monthly visits (348.6 M, 47.8 M, and 20.2 M, respectively), while MOM and OER World Map received the least (5 K and 5.1 K). Similarly, Commons, MIT OCW, and Open Textbook Library received the most unique visitors (22.1 M, 20.3 M, and 7.1 M, respectively), while MOM and OER World Map received the least (4.7 K and 3.2 K).

In most ROER, most traffic came from the USA: Commons (83%), Curriki (96%), MERLOT (77%), MOM (73%), OASIS (87%), OER Commons (78%), OpenStax (76%), and Open Textbook Library (82%) (Table 2). DOER received most traffic from India (28%), KlasCement from Belgium (83%), OER World Map from Germany (72%), and OpenLearn from UK (39%). It seems that DOER, MIT OCW, and OpenLearn have an international appeal since they received adequate traffic from various countries.

Most users visited directly Commons, Curriki, MIT OCW, OASIS, OER Commons, OpenStax, and Open Textbook Library (Table 2). This may happen because their users were well acquainted with these ROER and they frequently visit them. In DOER, KlasCement, MERLOT, OER World Map, and OpenLearn, most visits came via Search Engines. It was unexpected that most users did not visit directly MERLOT and OpenLearn although these popular ROER have easy to remember URL (Universal Resource Locator). Finally, most visits to MOM (which has a difficult to remember URL) came from referrals from other websites.

Visitors to Commons, OASIS, Open Textbook Library, and MIT OCW viewed the highest number of pages per visit (8.45, 6.15, 5.78, and 5.54, respectively), while visitors to Curriki, DOER, and OpenStax the lowest (1.09, 1.71, and 1.78, respectively) (Table 2). The average visit duration was the longest in Commons (26 min), Open Textbook Library (18:31), OASIS (16:50), and MIT OCW (15:44), while it was the shortest in DOER (2:28) and MOM (4:08) (Table 2). Finally, the lowest (best) bounce rate was achieved by Commons (18%), while the highest (worst) by Curriki (91%), OER World Map (79%), and OpenStax (72%) (Table 2).

Reputation & Authoritativeness of the 13 ROER

Regarding the ROER’ Reputation & Authoritativeness, Semrush showed (Table 3) that the highest authority score was achieved by MIT OCW (85) followed by Commons (80) and Open Textbook Library (79). The lowest authority score goes to MOM (38) and OER World Map (38).

Most referring domains pointed to MIT OCW (50.27 K), MERLOT (13.5 K), OER Commons (10.09 K) and OpenStax (9.4 K) (Table 3). Most backlinks pointed to KlasCement (11 M), MIT OCW (9 M), OER Commons (5.8 M), and MERLOT (5.1 M). Note that Google considers a website more trustworthy if it receives many backlinks from many different domains. Klascement received many backlinks (11 M) but these backlinks came from a few domains (3.3 K).

On the contrary, the lowest number of referring domains pointed to DOER (97), OpenLearn (191), MOM (330), and OASIS (467) (Table 3). Also, few backlinks pointed to DOER (1.2 K), OpenLearn (1.7 K), and OASIS (8.3 K). It is strange that the well-known OpenLearn was referred by only 191 domains with 1.7 K backlinks. However, Alexa showed that about 1.7 K websites pointed to OpenLearn (Table 4). It was observed that there was a difference between the measurements by Semrush and Alexa. This is not a rare case since these Web Analytics tools use different methods to measure various metrics. Actually, each tool may not give the “absolute real number” for each metric for a given website but rather a relative number in comparison with other websites for this metric. Note also that the numbers of referring domains and backlinks to most ROER are growing rapidly day-by-day. So, these numbers will be very different in a few years.

All ROER had very low traffic coming from mobile devices (Table 3). OpenLearn had the highest percentage of traffic coming from mobile devices (50%), while OER World Map (0%), MOM (5%), and Commons (9%) had the lowest.

Using SpyFu to measure the ROER Reputation showed (Table 4) that MIT OCW achieved the best rank (15.1) followed by OpenLearn (17.2), OER Commons (18.1), and OpenStax (19.7) (lower ranking number corresponds to better ranking position). MOM and OASIS were not even ranked due to their very bad ranking (> 51). Almost all ROER improved their ranking position over the last 3 years (Fig. 2).

Fig. 2

Improvement of ROER ranking over time. Data provided by SpyFu (when this information is available)

Using Alexa to measure the ROER Reputation (Table 4) revealed that the highest number of linking sites pointed to MIT OCW (65.4 K) followed by Open Textbook Library (27.4 K) while the lowest number of linking sites pointed to OER World Map (7), OpenStax (100), and Curriki (276). Note again that Alexa showed different results than Semrush. For example, the following different numbers were shown by Alexa (linking sites) versus Semrush (referring domains) for KlasCement (258 versus 3.3 K), MERLOT (1.4 K versus 13.5 K), OER Commons (759 versus 10 K), OpenLearn (1.6 K versus 191), and OpenStax (100 versus 9.4 K). It seems that Alexa underestimated the number of websites pointing to a ROER except for the case of OpenLearn.

The number of daily page views per visitor was the highest for Commons (10.6) followed by Klascement (4.3) and MIT OCW (4.12) (Table 4). Similarly, the longest daily time on site belonged to Commons (12:17) followed by MIT OCW (5:41). On the contrary, the number of daily page views per visitor was the smallest for OASIS (1.4), Curriki (1.8), and OpenLearn (1.8). Similarly, the daily time on site was the smallest for Curriki (1:06), OASIS (1:45), and OpenLearn (2:22). Regarding the bounce rate, Commons achieved the best (16%) while OASIS (78%), Curriki (75%), and OpenLearn (72%) the worst.

Accessibility of the 13 ROER

The accessibility of these ROER was tested using five Web Accessibility tools: (1) AChecker (WCAG 2 AA) [1], (2) ANDI (Accessible Name & Description Inspector) Accessibility Testing Tool [3], (3) EqualWeb A11y Checker [19], (4) Siteimprove Accessibility Checker [43], and (5) Web Accessibility [47]. These tools gave mixed results (Table 5). Although the Web Accessibility tool showed that all of these ROER achieved high accessibility scores above 91%, some ROER exhibited many violations (e.g., 35 violations by MIT OCW). According to Achecker (WCAG 2 AA), Curriki, MERLOT, MIT OCW, and OER Commons exhibited many ‘Known problems’ that have been definitely identified as accessibility obstacles. According to ANDI Accessibility Testing Tool, some common accessibility issues included the following: ‘Keyboard Access Alerts,’ ‘Elements with No Accessible Name,’ ‘ARIA-Hidden Alerts,’ ‘Misuses of Alt attribute.’ According to EqualWeb A11y Checker, Curriki and MIT OCW had many errors that violate WCAG. ROER administrators should urgently fix them since they may affect users with disabilities. According to Siteimprove Accessibility Checkers, some common accessibility issues in the examined ROER included the following: (1) ‘Color contrast is not sufficient,’ (2) ‘Text is clipped when resized,’ (3) ‘Text not included in an ARIA landmark,’ (4) ‘Visible label and accessible name do not match,’ (5) ‘Hidden element has focusable content,’ (6) ‘Line weight is below minimum value,’ and (7) ‘Links are not clearly identifiable,’ It seems that DOER, OASIS, OpenStax, and OpenTextbook Library were the most accessible ROER. However, at least three Web Accessibility tools identified several accessibility problems in Curriki and MIT OCW (Table 5).

Table 5

Measuring ROER accessibility using Siteimprove accessibility checker, ANDI accessibility testing tool, EqualWeb A11y Checker, AChecker (WCAG 2 AA), and web accessibility

Repositories of OER (ROER)	Siteimprove Accessibility Checker: Number of Issues: Specific Issues	ANDI: Focusable Elements; Accessibility Alerts	EqualWeb A11y Checker: General Errors; Contrast Errors; Notices; Warnings; Aria Attribute; Role attribute	AChecker (WCAG 2 AA): Known Problems; Likely Problems; Potential Problems	Web Accessibility: Health Score; Violations
Commons	4: Headings are not structured → 1 Container element is empty → 1 Color contrast is not sufficient → 43 Text is clipped when resized →  + 99	107; 49. Elements with No Accessible Name → 1; Keyboard Access Alerts → 48	0; 97; 128; 160; 407; 300	2; 0; 13	100%; 0
Curriki	10: Link without a text alternative → 6 Button without a text alternative → 1 Headings are not structured → 1 Text not included in an ARIA landmark → 4 Links are not clearly identifiable → 1 Color contrast is not sufficient → 3 Line height is below minimum value → 4 Font size is fixed → 2 Text is clipped when resized → 12 Presentational element is exposed to assistive technologies → 1	36; 7. Elements with No Accessible Name → 7	44; 4; 229; 31; 6; 4	101; 0; 527	93%; 9
DOER	5: Image without a text alternative → 1 Headings are not structured → 1 Text not included in an ARIA landmark → 68 Container element is empty → 1 Line height is below minimum value → 3	12; 0	12; 4; 107; 12; 3; 2	7; 0; 152	96%; 2
Klas Cement	3: Link without a text alternative → 3 Text not included in an ARIA landmark → 2 Line height is below minimum value → 8	46; 0	6; 0; 99; 19; 59; 31.	24; 0; 334	92%; 7
MERLOT	5: Visible label and accessible name do not match → 8 Element IDs are not unique → 4 Headings are not structured → 1 Text not included in an ARIA landmark → 2	?	1; 6; 172; 50; 93; 34.	79; 1; 226	96%; 5
MIT OCW	9: Link without a text alternative → 1 Inline frame without a text alternative → 1 Hidden element has focusable content → 23 Element IDs are not unique → 5 Headings are not structured → 1 Text not included in an ARIA landmark → 18 Links are not clearly identifiable → 11 Line height is below minimum value → 43 Text is clipped when resized → 1	167; 26. Elements with No Accessible Name → 2; ARIA-Hidden Alerts → 23; Reference Alerts → 1	23; 0; 120 107; 93; 26	52; 0; 294	91%; 35
MOM	6: Button without a text alternative → 3 Page language has not been identified → 1 Headings are not structured → 1 Text not included in an ARIA landmark → 5 Line height is below minimum value → 1 Form field is not labeled → 2	67; 10. Elements with No Accessible Name → 5; Excessive Text → 3; Small Clickable Areas → 2	18; 0; 112; 11; 70; 9	0; 0; 0	93%; 8
OASIS	3: Text not included in an ARIA landmark → 12 Container element is empty → 1 Form field is not labeled → 1	21; 0	1; 0; 92; 19; 8; 2.	12; 0; 137	100%; 0
OER commons	14: Inline frame without a text alternative → 2 Hidden element has focusable content → 1 State or property not supported → 1 Invalid state or property → 1 Image without a text alternative → 1 Element IDs are not unique → 2 Headings are not structured → 1 Text not included in an ARIA landmark →  Color contrast is not sufficient → 1 Line height is below minimum value → 2	63; 4. Aria-Hidden Alerts → 1; Elements with No Accessible Name → 2; Keyboard Access Alerts → 1	16; 2; 267; 27; 140; 80	68; 0; 311	91%; 10
OER World Map	6: Visible label and accessible name do not match → 16 Hidden element has focusable content → 1 Headings are not structured → 1 Text not included in an ARIA landmark → 74 Color contrast is not sufficient → 11 Form field is not labeled → 2	83; 11. Elements with No Accessible Name → 2; Aria-Hidden Alerts → 1; Misuses of Alt attribute → 6; Disabled Element Alerts → 1; Keyboard Access Alerts → 1	11; 26; 212; 20; 127; 31	30; 0; 166	96%; 3
Open learn	4: Text not included in an ARIA landmark → 5 Color contrast is not sufficient → 2 Line height is below minimum value → 27 Text is clipped when resized → 4	105; 30. Elements with No Accessible Name → 5; Keyboard Access Alerts → 25	14; 2; 234; 22; 13; 12	?	100%; 0
OpenStax	5: Element IDs are not unique → 6 Text not included in an ARIA landmark → 4 Color contrast is not sufficient → 1 Font size is too small → 2 Text is clipped when resized → 24	35; 0	3; 1; 122; 20; 36; 74	0; 0; 53	95%; 8
Open textbook library	3: Link without a text alternative → 3 Required ARIA attribute is missing → 1 Line height is below minimum value → 17	33; 5. Elements with No Accessible Name → 3; Aria-Hidden Alerts → 1; Keyboard Access Alerts → 1	3; 1; 134; 21; 36; 74	10; 0; 173	98%; 3

Conclusions, limitations, and future research

The thirteen ROER for teaching and learning differ with regards to their purposes, their target groups, their operation, the types of their curated OER, and many more characteristics. Subsequently, it is expected that they may differ with regards to their reputation and usage. So, this study does not try to rank them. Rather, it tries to picture their reputation, usage, and accessibility and possibly identify best cases. MIT OCW seems to be the most reputable ROER attracting millions of visitors and links pointing to its website. Commons also received millions of visitors and thousands of links pointing to its website. In many ROER, their visitors came from the country where the ROER hosting institute operates. However, visitors came to DOER, MIT OCW, and OpenLearn from a variety of countries. Actually, more than 60% of the visitors to OpenLearn came from outside UK. Similarly, more than half of the visitors to MIT OCW came from outside USA. Also, about one third of visitors to MERLOT, MOM, OER Commons, OpenStax came from outside USA. This means that all these ROER achieved to have an international appeal. In most ROER the visitors came directly to their websites. However, visitors came to DOER, KlasCement, and MOM after searching via a search engine, while they came to MOM after visiting other websites that refer to it. For some of the examined ROER (e.g., DOER, OpenLearn), Semrush did not show many domains and links pointing to their websites. So, it would be suggested to the administrators and managers of these ROER to try to increase their ROER visibility by implementing internet marketing campaigns and promoting their ROER to librarians, curators, educators, and learners.

Furthermore, since these ROER are not similar, there is no reason to calculate the averages and other statistics of their reputation or usage. Rather this study tries to outline the range of the ROER reputation and usage. So, their authority score extended from a high of 85 (MIT OCW) to a low of 38. The number of referring domains ranged from 100 to 50 K and the number of backlinks from 1 K to 11 M. However, the numbers of links pointing to the well-known ROER (e.g., MIT OCW) increase enormously day-by-day. So, it is expected that these numbers will be multiplied in the following years. The monthly visits ranged from 5 K to 350 M and the unique visitors from 3 K to 20 M. On average, the number of pages per visit ranged from 1.1 to 8.5 and the visit duration varied from 2 to 26 min, according to Semrush. Again, Commons and MIT OCW seem to be the most successful ROER achieving their visitors to stay for long time and visit many pages. The bounce rate extended from 18% (best, Commons) to 91% (worst). In some ROER, visitors visited only one page and left the website staying only for one (1) minute on average. The high bounce rate is not a good sign. So, administrators and managers of these ROER should try to enrich their ROER with quality, useful, and meaningful OER, support the visitors, and develop communities of curators, authors, experts, and teachers.

Although over 90% of internet users worldwide use a mobile device and the mobile internet traffic is over 54% of the total online traffic [45], it is strange that most ROER are not accessed via mobiles and only OpenLearn received half of its traffic from mobile devices. This does not necessarily mean that these ROER do not follow a mobile-friendly or responsive design. It may happen that their users do not prefer to search, access, and read OER in these ROER using mobile devices. Recognizing this reality, MIT OCW launched a new platform, NextGen, that offers a mobile-optimized environment [31]. It is expected that all ROER will soon recognize this widespread use of mobiles and redesign their websites following the paradigm of MIT OCW.

In general, Web Accessibility was good in most ROER. However, some of the ROER administrators and developers should intensify their efforts to closely monitor, audit, and fix the accessibility errors in their websites adhering to the World Content Accessibility Guidelines [50]. Using a combination of automated and semi-automated checks as well as user and expert manual testing, they can identify accessibility barriers and be supported in both automatically and manually correcting them.

In summary, a successful ROER attracts many visitors all over the world who stay and explore its website for long time. One way to achieve this is by curating many quality, useful, and accessible OER in various subjects and languages. Also, a successful ROER has a small bounce rate and a high authority score. In addition, a large number of quality domains and links point to its website. Furthermore, its website is mobile-optimized with no accessibility errors, violations, alerts, warnings, and other issues.

In order to achieve these aspects, the ROER should become well known, have an easy-to-remember URL as well as curate a large variety of useful, quality, and accessible OER for various educational disciplines, levels, languages, teaching and learning methods. Website developers should follow accessibility guidelines such as including alternative text for links and pictures, color contrast, flexible font size, structured headings, and more. Finally, the administrators and managers of the ROER should promote it to become well-known, visible, and reputable by implementing marketing campaigns and collaborations.

International collaboration among ROER, educational institutes, authors, teachers and others could mutually benefit all involved. More specifically, administrators and managers of ROER may agree to strengthen their collaboration and knowledge exchange. For example, OER curated in one ROER could be linked to relevant OER curated in other ROER. In this way, a ROER may boost its visibility and upgrade its services to users. In addition, a ROER may recommend appropriate OER (curated in its ROER as well as in other ROER) to users based on the users’ characteristics, preferences, needs as well as OER usage. A ROER could recommend to a user who is satisfied with a specific OER another similar OER. Also, a ROER could recommend to a user some OER that were used by similar users who were satisfied with these OER.

One of the limitations of this study is that it only used automated checks to assess the accessibility of the ROER websites. However, for a thorough accessibility assessment, user and expert-based testing should also be employed. Another limitation is that the measurements given by the Web Analytics tools are not always completely accurate. However, these measurements could be used as an indication of where a ROER stands with respect to other ROER for a specific metric. The findings could motivate the administrators and managers of a low performing ROER to upgrade the problematic areas of their ROER. Future research could also use other evaluation methods to measure the ROER reputation, authoritativeness, and usage as well as other quality parameters such as ROER usability and interoperability. Also, future research may investigate the reasons for which users do not stay for a long time at a ROER but leave it after visiting one or two pages. Finally, future research may investigate all reasons for which some ROER receive much traffic and hyperlinks pointing to them while others not.