COMP Report: CPQR Technical Quality Control Guidelines for Data Management Systems.

The Canadian Organization of Medical Physicists (COMP), in close partnership with the Canadian Partnership for Quality Radiotherapy (CPQR) has developed a series of Technical Quality Control (TQC) guidelines for radiation treatment equipment. These guidelines outline the performance objectives that equipment should meet in order to ensure an acceptable level of radiation treatment quality. The TQC guidelines have been rigorously reviewed and field tested in a variety of Canadian radiation treatment facilities. The development process enables rapid review and update to keep the guidelines current with changes in technology (the most updated version of this guideline can be found on the CPQR website). This particular TQC details recommended quality control testing of radiation data management systems.

INTRODUCTION

The Canadian Partnership for Quality Radiotherapy (CPQR) is an alliance among the three key national professional organizations involved in the delivery of radiation treatment in Canada: the Canadian Association of Radiation Oncology (CARO), the Canadian Organization of Medical Physicists (COMP), and the Canadian Association of Medical Radiation Technologists (CAMRT). Financial and strategic backing is provided by the federal government through the Canadian Partnership Against Cancer (CPAC), a national resource for advancing cancer prevention and treatment. The mandate of the CPQR is to support the universal availability of high quality and safe radiotherapy for all Canadians through system performance improvement and the development of consensus‐based guidelines and indicators to aid in radiation treatment program development and evaluation. The development of the individual TQC guidelines is spearheaded by expert reviewers and involves broad stakeholder input from the medical physics and radiation oncology community.1

This document contains detailed performance objectives and safety criteria for Data Management Systems. Please refer to the overarching document Technical Quality Control Guidelines for Canadian Radiation Treatment Centers2 for a programmatic overview of technical quality control, and a description of how the performance objectives and criteria listed in this document should be interpreted.

All information contained in this document is intended to be used at the discretion of each individual center to help guide quality and safety program improvement. There are no legal standards supporting this document; specific federal or provincial regulations and license conditions take precedence over the content of this document.

SYSTEM DESCRIPTION

The fundamental definition of a data management system (DMS) has not changed since the publication of the Canadian Association of Provincial Cancer Agencies (CAPCA) quality control document for data management systems in 2008: a DMS is the information infrastructure which is directly related to the planning, delivery, quality assurance, and archival of patient treatments. In its simplest incarnation, a DMS can be a single computer. However, in the typical radiation treatment clinic, a DMS is comprised of many separate entities or systems that manage, store, and exchange information of many types and formats via various methods and protocols. The level of complexity of computer systems in radiation oncology clinics has seen a tremendous increase over the past several years. In part, this increase is due to the evolution of radiation treatment technology — the increasing complexity of treatment delivery systems which themselves contain multiple computerized systems, the ongoing evolution of onboard imaging systems, the increased variety and quantity of diagnostic and simulation imaging studies involved in the planning process, and the ever‐increasing evolution and scope of record/verify electronic medical record systems. The ongoing transition of many clinics toward a paperless or “paper‐light” environment is even further increasing the quantity and variety of data stored and managed by computerized systems in the clinic. And of course, beyond the walls of our clinics, the world of information technology and data management is expanding at a relentless pace — so that the hospital infrastructure and systems that often form the backbone of our radiation clinics’ data management systems are also evolving rapidly.

A comprehensive quality assurance program for a DMS should consider all of the separate components in the DMS, the exchange of data between components, and the procedures governing that exchange. Accordingly, the program could have three general categories:

Quality assurance of computerized systems: performance and functionality of each individual component in the DMS, data integrity within each component;

Quality assurance of data exchange: data exchange between components in the DMS (multiple formats, multiple protocols, via interface, or manual data transfer); and

Quality assurance of procedures (including data entry and data interpretation).

Key features of a quality assurance program should include: assembling a multidisciplinary team with regular meetings and clearly established roles and responsibilities; project management of scheduled upgrades and systematic tracking and evaluation of hardware and software failures and issues, and subsequent root‐cause analysis.

Each radiation treatment clinic's DMS is unique, making it impossible to prescribe a universal or one size fits all quality assurance program. Instead, this guidance document offers a step‐by‐step approach to aid the medical physicist in designing a tailored, context‐specific quality assurance program for each unique DMS (see Appendix 1). The lists of test categories included in Tables 1 and 2 and the specific tests detailed in Section 5 are meant to be comprehensive but not prescriptive, serving as a recipe box from which the qualified medical physicist can select the appropriate tests for their unique DMS. Furthermore, testing frequencies must be established based on in‐depth knowledge of the relevant clinical processes — the suggestions made in this document serve as a reasonable baseline that should be modified to suit a given DMS. Some of the tests chosen for the DMS quality assurance program will likely be the responsibility of IT personnel. Others will be the responsibility of the medical physicist. It is probable that some of the tests will require collaboration and input from the appropriate vendor. The approach described here is adapted in part from that suggested by Report 93 of the Institute of Physics and Engineering in Medicine (IPEM),3 incorporating suggestions from the preliminary recommendations of the American Association of Physicists in Medicine's (AAPM) TG 2014, 5 as well as the existing CAPCA document for Data Management Systems. While not entirely in scope, this document may provide some guidance for those who are in the process of developing a new center's DMS or merging an existing DMS within an existing hospital's IT infrastructure.

Table 1

Categories of quality control tests for DMS data links, components and procedures

Designator	Test	Performance
DMS data links
L1	Data transfer integrity	Complete
L2	Data transfer integrity of images and imaging data	Complete
L3	Data transfer integrity of electronic documents	Complete
L4	Tests for failure	Complete
L5	Soak or endurance testing	Complete
DMS components
C1	Performance tests	Complete
C2	Network tests	Compare to baseline
C3	Security tests	Complete
C4	Data integrity	Complete
C5	Tests for failure	Complete
C6	Machine readout checks	Complete
C7	Data capture	Complete
C8	General administrative checks	Complete
Procedures
P1	End‐to‐end testing	Complete
P2	Review of clinical process maps	Complete
P3	Contingency plan review	Complete

Table 2

Overall quality control program tests for a general DMS similar to Figs. A1, A2, A3, A4, A5, A6

Designator	Test	Performance
Annual overall program
O1	Quality assurance program review	Complete
O2	Review of hardware and software inventories and DMS system maps	Complete
O3	Review of audits (network traffic, user access, etc.)	Complete
O4	Scheduled upgrades	Complete

Figure A1

An example of components in a single site DMS (simple version)

Figure A2

An example of components in a single site DMS (refined version)

Figure A3

Single site DMS example: Clinical data flow.

Figure A4

An example of a multisite DMS configuration.

Figure A5

DMS System Map including data types (I: Images & imaging data, T: Treatment parameters, G: General/demographics, D: Electronic documents) and methods of data transfer (M: Manual, RM: Removable media, HL7: HL7 interface, DCM: DICOM or DICOM RT, CI: Custom interface, PI: Proprietary interface, Net: Intranet).

Figure A6

An example of a multisite DMS: System Map.

In many clinics, IT personnel will be responsible for physical networks and hardware, as well as existing software and hospital information management systems. Consequently, it is entirely possible that the medical physicist responsible for a radiation treatment clinic's DMS will not have the necessary resources, knowledge and/or access to design and implement the DMS quality assurance program on their own. In a comprehensive report,3 the IPEM highly recommends that management of the radiation treatment DMS be maintained by medical physicists, with the support of dedicated IT specialists. Another model requires responsible medical physicists with solid IT skills to act as application owners for all “medical device tier” systems — those clinical systems directly affecting patient care. The maintenance of physical and virtual networks and computers and associated software falls under the responsibility of IT personnel. For safe and effective delivery of radiotherapy, a high degree of collaboration and a certain degree of knowledge‐overlap is needed between the responsible medical physicist and IT personnel. This requires the assigned IT support staff to be on site.4 For further guidance regarding appropriate training for radiotherapy IT professionals, see Siochi et al., 2009, Information Technology Resource Management in Radiation Oncology.4

It is recognized that existing organizational structures vary widely between individual radiotherapy clinics. While in some cases, the IT specialists are members of the radiotherapy department, this is certainly not always the case. It is recommended that, as a minimum, consultation and ongoing collaboration with responsible IT specialists will be required. A multidisciplinary team should be established to share responsibility for the DMS quality assurance program. The roles and responsibilities of all team members must be clearly defined to ensure accountability.5 This collaboration will ensure that responsible IT personnel are aware of the details, complexities, and critical nature of the systems used for radiotherapy as well as ensure appropriate resources for management, testing, maintenance, security, trouble‐shooting, training, and support.3

TESTING TOOLS

Software

While many of the tests suggested here can be executed manually using test data, certain categories of tests are better suited to automated testing, specialized testing tools exist that have the capability to manage, script, and automate soak or endurance tests, network performance tests, and/or data integrity checks. Examples include, but are not limited to, Microsoft Visual Studio Testing Tools and Services, AgileLoad, HeavyLoad, Solarwinds, and Inflectra SpiraTest. Some tools are designed for specific types of testing, whereas other more comprehensive software suites offer tools to script and execute tests based on a schedule and include project management components to store and track the results of each test.

In some cases, manual end‐to‐end testing with a large number of test scenarios can be resource‐intensive. Automated testing tools are available that mimic a user's interaction with a workstation by sending data as if a user were interacting with the environment. A set of test data can be constructed and should be chosen to include the range of clinically relevant scenarios. Network communication from a sending computer to a receiving computer can be replicated. Some testing software will record a user‐interaction and store it as a test script, which can then be modified and/or replicated. Certain types of tests cannot be fully automated and require user interaction. For these manual tests, the software can prescribe the specific steps a tester is required to follow, including test data and order, state the expected behavior at each step and allow the tester to document the results.

The integration of such testing tools in a clinic's DMS requires collaboration with IT personnel and the appropriate vendors. Also note that the timing of automated tests should include sufficient delays between executions to avoid placing an artificial load on the system (which could artificially produce errors).

Checksums and data validation

A checksum is a type of redundancy check that can be used to evaluate data integrity following transmission across a network (or data link), or following any other manipulation that could introduce error. An algorithm is used to calculate binary or other values that represent the data packet that can be compared at the beginning and end of each test point. If the outputted values are not identical, then the data being tested have changed in some way. Checksums algorithms can be executed against various data, including very large data items, which are otherwise difficult or time‐consuming to compare. Usually using a cryptographic hash (or similar) function, given a specific data value, it will always produce exactly the same result. Further, there are other approaches that operate similarly to checksums, for example, Cyclic Redundancy Codes (CRC). Each has its benefits and weaknesses and the choice of tools must be evaluated based on reliability, cost, and criticality of the data/component/link being tested. These data validation methods are not usually necessary on a routine basis, but are useful when verifying the integrity of user‐specific data, such as treatment planning beam models during upgrades, software revisions, and maintenance releases.

Virtualizations

Generally, virtualization refers to the creation of a virtual version of a device, such as a server, network storage device, operating system, application, or service. Virtualization should only be performed with the support of the appropriate vendors as not all DMS components lend themselves to virtualization. In some cases, DMS performance can be negatively impacted.

When testing virtualized environments such as, but not limited to Citrix virtualization, the unique architecture of these services needs to be taken into account. These environments reduce the need to manage many applications and/or desktop environments across multiple workstations and operating systems and thus allow for workstations that span multiple network security domains (or other complex configuration differences) to access the applications or possibly entire virtual desktops through a single managed network port and protocol. Application virtualization relies on a locally installed receiver application that communicates from the remote workstation to the application server, where the application is running. Modern receiver applications generally are add‐ons to the workstation's web browser.

Virtualized environments require special testing considerations. The interface presented to users is virtualized — the user is presented with an image of the user interface of an application or desktop that is running on a remote server. This can introduce latency and lead to possible data input errors from the user, particularly in graphically intensive tasks such as contouring, registration, and dose display and manipulation. Virtual environments add complexity to automated tests that are meant to replicate user input — as most of these tests are configured to input data and submit as if from a user's workstation. Tests can be operated from within a virtual desktop through the virtualization services, but this does not replicate data passing through the “screen scraper” running on a local workstation.

Since remote applications are running from a server (virtualized or not), soak or endurance testing (see Designator L5 in Table 1) against the virtualization service is important — as if this service falters or fails, then any application in the DMS provided through the virtualization service may be impeded or unavailable.

When testing for failure (see Designator L4 in Table 1), the effect of a session timeout on an application when a user attempts to reconnect should be investigated. What is the state of the session and its data after the connection times out and is it recoverable? For the majority of applications on a modern virtualization environment, sessions should be recoverable for reconnection, within some period of time, and ideally harmonized with security and automatic disconnection settings (e.g., screen saver settings, automatic log‐off after inactivity).

In virtualized environments, imaging data are often compressed for faster data transfer. This compression and decompression can lead to reduced image quality. It is recommended that particular attention be paid to tests of data transfer integrity of images and imaging data (Table 1, Test designator L2).

Modern DMS infrastructures now virtualize servers as they offer lower cost of ownership, reduced footprints (hence reduced power and cooling requirements), more efficient data migration and backup/restores, and easier customization of hardware. They do, however, require additional layers of maintenance (hypervisors) and management which can affect overall DMS performance. Because they are often hosted through shared storage (SAN) environments not necessarily within the confines of the radiation oncology treatment center or location, they are susceptible to data loss due to a catastrophic events, large‐scale system degradation and outages. While the management of virtualized servers falls within the scope of IT professionals, given the risks to the integrity of the RO‐IT infrastructure, changes to virtualized server settings should be communicated to the DMS multidisciplinary team.

RELATED TECHNICAL QUALITY CONTROL GUIDELINES

In order to comprehensively assess data management system performance, additional guideline tests for integrated systems, as outlined in related CPQR Technical Quality Control (TQCs) guidelines must also be completed and documented, as applicable. Related TQC guidelines,6 available at cpqr.ca, include:

CT Simulators

Treatment Planning Systems

Medical Linear Accelerators and Multileaf Collimators.

TEST TABLES

Notes on tests for DMS links

Tests in this section are applicable to each data link (joining two computerized systems within the DMS). In addition to the tests suggested here, vendor recommendations for commissioning, acceptance testing, and regular quality assurance should be followed.

Data transfer integrity of general/demographics and treatment parameter data

Test: For an appropriate range of clinical data, compare data sent vs. data received. Manual or automated tests can be performed using checksums or custom scripts using a bank of test data.

Tolerances: Different systems may have different levels of accuracy and also may have differing naming conventions. This can lead to errors — for example, due to rounding or truncation of data. Tolerances need to be established (whether zero or non zero) wherever data transfer occurs. To facilitate data transfer integrity tests, it is very helpful to construct data transfer tables for each data link. A data transfer table should include a full list of all parameters transferred, and the tolerances associated with the transfer. It is important to also be aware of any differences between the internal format or convention of the raw data and that displayed to the user. Data dictionaries can be valuable resources in the construction of these tables. Note that the selection of an appropriate range of clinical data is a nontrivial task requiring careful consideration of all clinically relevant treatment scenarios. A library of test cases can be constructed in the treatment planning system for use as needed and should be updated to reflect new and emerging treatment scenarios.

Suggested frequency: At commissioning, and following any change to the DMS components connected by the data link that could affect clinical data (including data formats, storage, display, tolerances, transfer protocols, etc.). A data transfer integrity test may be appropriate as part of routine patient quality assurance for certain clinical protocols — though it is likely this test will be of more limited scope. An example could be to compare critical treatment data given by the treatment console against a screen capture of approved plan data.

Data transfer integrity of images and imaging data

Geometric integrity (scale, accuracy)

Test: Use a geometric test phantom with known dimensions and compare data before and after transfer, including appropriate scaling and/or processing.

Suggested frequency: At commissioning, and following any change to the DMS components connected by the data link that could affect imaging data (e.g., upgrade of cone beam CT [CBCT] software). This test is often part of existing quality assurance of imaging systems.

Coordinate frame and patient orientation

Test: Use a test phantom whose orientation is clearly identifiable. For all relevant orientations and positions, confirm that images are correctly transferred and interpreted.

Suggested frequency: At commissioning, and following any change to the DMS components connected by the data link that could affect imaging data (e.g., upgrade of CBCT software). This test is often part of existing quality assurance of imaging systems.

Data transfer integrity of images and imaging data

Test: Image quality: Using an appropriate phantom, evaluate image contrast, noise, and image intensity (e.g., HU value). Identify data degradation or distortion (e.g., due to compression). Compare values before and after image transfer. Compare against baseline or tolerance values as appropriate.

Test: File integrity: Using checksums or other tools, evaluate the integrity of the imaging files before and after transfer. Note that this test is required in addition to the above tests as it is possible for errors in integrity to be introduced that will not be visually apparent or detectable within the software used for image analysis.

Suggested frequency: At commissioning, and following any change to the DMS components connected by the data link that could affect imaging data (e.g., upgrade of CBCT software). This test is often part of existing quality assurance of imaging systems.

Data transfer integrity of electronic documents

Test: Verify that transfer of electronic documents occurs as expected and that data format and integrity is maintained. Test should include all relevant document formats. Checksum or appropriate tools should be used in addition to visual inspection as errors can be introduced that will not inhibit document processing software from opening and manipulating the file.

Suggested frequency: At commissioning, and following any change to the DMS components connected by the data link that could affect the transfer of electronic documents [e.g., upgrade of electronic medical record (EMR) software].

Tests for failure

Out of limits data handling

Test: Check that data exceeding limits of the receiving system are appropriately handled. This could include extra decimal points, null values, clinically irrelevant gantry or collimator examples, etc.

Data corruption

Test: Where possible, simulate data corruption and evaluate how this is handled by the receiving system. Include the handling of missing or truncated data, as may occur when data transfer is interrupted. Vendor input may be required to identify other relevant scenarios.

System recovery from error state

Test: Where possible, simulate error states and check that systems recover appropriately (and check for proper handling of data/data recovery). If mitigating steps are required to recover data, include these in your contingency plans. For example, some linear accelerators require the user to manually recover the record of delivered monitor units (MU) when a treatment beam is interrupted by a power failure or other unexpected system shut‐down.

Where possible, tests A, B, and C should be automated to allow for large sets of sample data to be evaluated, recording the results of each execution and comparing against expected behavior.

Suggested frequency L4 A–C: At commissioning, and following any change to the DMS components connected by the data link that could affect clinical data (potentially use an appropriate subset of tests depending on the change to the system).

Soak or endurance testing

Test: Test the ability of the system to handle large volumes of clinical data within a short time period. This test should be automated using testing tools as described later in this document. Soak or endurance testing should be performed after a full system backup and should not be performed during clinical operation. A set of interactions and inputs that reflect standard user input across the data link can be scripted. These are replicated in increasing test sizes so that the load on the system is stepped up over a measured time period. Performance counters should be collected during this time until the environment becomes unresponsive or until foreseeable data loads have been exceeded. Soak or endurance testing can be combined with data integrity tests as outlined in L1–L3 as data integrity results may be impacted by a higher system load, even if the environment appears to be operating properly (and is not yet unresponsive).

Suggested frequency: At commissioning or as needed (for example during troubleshooting for performance issues).

Notes on tests for DMS components

Tests in this section apply to individual computerized systems within the DMS. In addition to the tests suggested here, vendor recommendations for commissioning, acceptance testing, and regular quality assurance should be followed. Some tests are also applicable to DMS data links — they are listed again here for completeness as they should be considered when implementing a new DMS component (or following an upgrade or other significant change).

Performance tests

Test: Check accuracy of data transfer (using checksums, automated data transfer, redundancy checks, etc.) (see L1–L3 for details and suggested frequency).

Test: Monitor delay times (changes from baseline)

Suggested frequency: At commissioning and on an ongoing basis. Baseline values and thresholds should be established with the collaboration of the responsible IT personnel with input from vendors as appropriate.

Test: Monitor available memory, CPU usage (set thresholds)

Suggested frequency: At commissioning and on an ongoing basis. Lack of available memory can have unexpected impacts on performance and could lead to errors in data integrity. Automated tools exist to monitor system resources and alert system administrators when an established threshold is reached. If automated tools are not available, close monitoring is required.

Network tests

Test: Validate accuracy and completeness of clinical data

Suggested frequency: At commissioning and quarterly.

Test: Monitor routing tables or routed and gated daemons.

Test: Connectivity tests [Digital Imaging and Communications in Medicine (DICOM), Echo, ping checks] between DMS components

Test: Monitor functionality of required services (depending on operating system)

Test: Monitor network speed/resource allocation (time required to transfer large test data between servers or computers, lag in console display)

Suggested frequency: C2 B–E: At commissioning, on an ongoing basis and during troubleshooting (of connectivity issues, for example). Baseline and threshold values depend strongly on the network design and infrastructure and should be established in collaboration with qualified IT personnel. Automated tools exist to monitor many aspects of network performance against established thresholds.

Security tests

Test: Check for manufacturer security fixes (unless automatically provided by vendor)

Test: Maintain up‐to‐date list of applications, versions, patches, service packs, operating systems, etc.

Test: Maintain up‐to‐date antivirus software

Test: Adherence to pushed antivirus and other policy settings for standard and nonstandard computers

Test: Appropriateness of virus scan settings on specific workstations and servers (real‐time vs. scheduled for critical workstations and servers)

Test: Monitor user and system logs

Test: Evaluate and monitor physical and network boundaries including firewall settings

Test: Control user access permissions

Test: Physical hardware checks

Suggested frequency: C3 A–I: At commissioning and on an ongoing basis or as needed. Many of these tests may fall within the responsibility of IT personnel or will require input from IT personnel and/or vendors. Radiotherapy system components are highly sensitive to changes to security settings (virus scan settings, for example). The resulting loss of performance can be very difficult to troubleshoot without input and ongoing communication with IT personnel.

Data integrity

Test: Validate accuracy and completeness of clinical data

Suggested frequency: Weekly or as appropriate based on criticality of data

Test: Data transfer tests (see Table 1 DMS Data Links tests)

Test: Automated data transfer checks (checksums, etc.)

Test: Validate backup/restore functionality (if applicable)

Suggested frequency: C4 A–D: At commissioning and following any change to the DMS component that could affect clinical data.

Tests for failure

Test: Soak/endurance tests (see L5)

Suggested frequency: At commissioning

Test: Evaluate performance of critical systems following failure or simulated error states (system recovery, record locking, etc.).

Suggested frequency: At commissioning and following any change to the DMS component that could affect clinical data or on a regular basis where appropriate.

Machine readout checks

Test: Compare display/readout to database. Note that the data visible to the user may be in a different format or following a different convention from that stored internally. Such conversions must be evaluated to ensure correct representation of data and appropriate degree of accuracy

Suggested frequency: At commissioning and following any change to the DMS component that could affect clinical data

Data capture

Test: Validate capture of clinical data following user input. For example, couch settings as captured by treatment unit console and/or control software could be compared to treatment table readouts

Suggested frequency: At commissioning and following any change to the DMS component that could affect clinical data capture

General administrative checks

Test: Check format/accuracy of printouts or other. Data included in printouts may follow a different format or convention and must be compared to ensure accuracy and correct representation

Suggested frequency: At commissioning and following any change to the DMS component that could affect clinical data

Categories of tests for DMS procedures

End‐to‐end testing

Test: Using carefully constructed, clinically relevant test cases, validate the complete clinical data chain from simulation to dose delivery. Test cases must be chosen to cover the full range of possible clinical situations.

Suggested frequency: At commissioning or following a change to any component of the DMS that is part of the routine clinical data flow. This type of testing is also valuable as part of the validation of a new treatment technique or, for some clinical protocols, as part of patient quality assurance. Regular end‐to‐end testing may be appropriate, especially in large systems with shared responsibility and management where changes to the DMS may occur without the responsible physicist's knowledge.

Note that end‐to‐end testing alone is not sufficient — though the test result may show an error, it will not necessarily identify the source or cause of the error. In addition, end‐to‐end testing relies on test case construction. Without full testing of data transfer integrity between components in the DMS as outlined above, it is entirely possible to miss errors that will later impact clinical data.

Review of clinical process maps

Test: Review existing documentation of clinical processes and update to reflect changes to DMS system components, links and/or procedures. Ideally, this test should be executed by a multidisciplinary team responsible for the DMS quality assurance program.

Suggested frequency: Annually or following a change to a DMS component that is part of the routine clinical data flow.

Contingency plan review

Test: Review contingency procedures

Suggested frequency: Annually or following a change to backup and recovery systems within the DMS. Ideally, this test should be executed by a multidisciplinary team responsible for the DMS quality assurance program.

Test: Test alternative clinical data pathways.

Suggested frequency: Annually or following a change to backup and recovery systems within the DMS (during planned outages where possible).

CONFLICT OF INTEREST

The authors have no conflicts of interest.

Table A1

Categorization of the importance and risk of data flow elements for the single site DMS example featured in Figs. A1, A2, A3, A4, A5, A6

Source	Receiver	Data Types	Method(s)	Importance	Risk of Failure
	CTSim	General	Manual	High	High
MOSAIQ	PRIMUS (SEQUENCER)	General, image, treatment parameters	Proprietary interface	High	Medium
PRIMUS (SEQUENCER)	MOSAIQ	Image, treatment parameters	Proprietary interface	High	Medium
PRIMUS (SEQUENCER)	Coherence Therapist	General, Image, Treatment Parameters	Proprietary Interface	High	Medium
Coherence Therapist	PRIMUS (SEQUENCER)	General, image, treatment parameters	Proprietary interface	High	Medium
PACS	Pinnacle	Image	DICOM	High	Medium
CTSim	Pinnacle	Image, general, treatment parameters	DICOM, DICOMRT	High	Low
Pinnacle	MOSAIQ	Image, treatment parameters, documents	DICOM, DICOMRT, intranet	High	Low
MOSAIQ	SYNERGY (SEQUENCER)	General, image, treatment parameters	Proprietary interface	High	Low
SYNERGY (SEQUENCER)	MOSAIQ	Image, treatment parameters	Proprietary interface	High	Low
SYNERGY (SEQUENCER)	SYNERGY (LCS)	Treatment parameters	Proprietary interface	High	Low
SYNERGY (LCS)	SYNERGY (SEQUENCER)	Treatment parameters	Proprietary interface	High	Low
Coherence Therapist	PRIMUS (LINAC CONTROL)	Treatment parameters	Proprietary interface	High	Low
PRIMUS (LINAC CONTROL)	Coherence Therapist	Treatment parameters	Proprietary interface	High	Low
	Meditech	General	Manual	Medium	High
MOSAIQ Data Director	MOSAIQ	Image	DICOM	Medium	Medium
Meditech	MOSAIQ	General (Demo)	HL7 interface	Medium	Low
CTSim	Brachyvision	Image	DICOM	Medium	Low
SYNERGY (SEQUENCER)	XVI	general, image	Proprietary interface	Medium	Low
XVI	SYNERGY (SEQUENCER)	Image, treatment parameters	Proprietary interface	Medium	Low
SYNERGY (SEQUENCER)	iView	General, image	Proprietary interface	Medium	Low
iView	SYNERGY (SEQUENCER)	Image, Treatment parameters	Proprietary interface	Medium	Low
Pinnacle	Coherence Oncologist	Image	DICOM	Medium	Low
Coherence Oncologist	Coherence Therapist	Image	DICOM	Medium	Low
MOSAIQ	Orthovoltage	Treatment parameters	Proprietary interface	Medium	Low
Orthovoltage	MOSAIQ	Treatment parameters	Proprietary interface	Medium	Low
MOSAIQ	CTAR	General	Custom Interface	Low	High
MOSAIQ	Meditech	General (schedule)	Custom interface	Low	Medium
MOSAIQ	MOSAIQ Data Director	Image	DICOM	Low	Medium
Brachyvision	MOSAIQ	Documents	Intranet	Low	Low

Table A2

Components of the DMS in Figs. A1, A2, A3, A4, A5, A6 — evaluation for inclusion in the DMS quality assurance program.

Component	Manufacturer	Notes	Relevant QA program or responsible party	Include in DMS QA program?
Meditech	Medical Information Technology, Inc.	Hospital information system (HIS)	Provincial IT
MOSAIQ	Elekta	R&V, EMR	Performance, network and security tests are the responsibility of Provincial IT, all other aspects to be addressed by DMS quality assurance program	✓
CTAR	Accreon	Provincial data repository	Provincial IT
Brilliance Big Bore CTSim	Philips		CTSim quality assurance program
Pinnacle (Hardware)	Philips	Pinnacle thin client solution — servers and compute modules reside in IT department	Hardware maintenance is responsibility of Philips
Pinnacle (TPS)	Philips	Treatment planning system	No existing quality assurance program — will be developed separately based on TQC for treatment planning systems1
Brachyvision	Varian Medical Systems	Brachytherapy treatment planning system	No existing quality assurance program — will be developed separately based on TQC for treatment planning systems1
GammaMed	Varian Medical Systems	Brachytherapy treatment machine	Hardware and software maintenance are the responsibility of Varian. HDR quality assurance program
Orthovoltage	XStrahl Ltd.	Orthovoltage treatment unit	Orthovoltage quality assurance program
SYNERGY (SEQUENCER)	Elekta	MOSAIQ station at treatment unit; communicates with linac control console via SYNERGISTIQ	Linear accelerator quality assurance program
SYNERGY (LCS)	Elekta	Linac control console providing input to Linac Control System (LCS), operates in “Receive External Prescription” mode to receive treatment parameters from SEQUENCER	Linear accelerator quality assurance program
SYNERGY (XVI)	Elekta	kVCBCT, communicates with SEQUENCER (including transfer of table parameters after imaging)	Linear accelerator quality assurance program
SYNERGY (iView)	Elekta	MV Portal Imaging (EPID), auto‐forwards images to MOSAIQ, imaging analysis in MOSAIQ (SEQUENCER)	Linear accelerator quality assurance program
PRIMUS (SEQUENCER)	Siemens	MOSAIQ station at treatment unit; communicates with linac control console via Coherence Therapist	Linear accelerator quality assurance program
PRIMUS (Coherence Therapist)	Siemens	Communicates with linac control console, receives treatment parameters from SEQUENCER and imaging data from Coherence Oncologist	No existing quality assurance program — to be added to Linear Accelerator quality assurance program
PRIMUS (Coherence Oncologist)	Siemens	Contouring station; receives images from Pinnacle, transfers images and contours to coherence therapist	No existing quality assurance program — to be added to Linear Accelerator quality assurance program
PRIMUS (Linac Control)	Siemens	Receives and updates treatment parameters with coherence therapist	Linear accelerator quality assurance program
MOSAIQ Data Director	Elekta	MOSAIQ Oncology Pacs system; archive of all DICOM image data for patients	Existing quality assurance Program

Table A3

Data flow table for the DMS in Figs. A1, A2, A3, A4, A5, A6 — evaluation for inclusion in the DMS quality assurance program.

Source	Destination	Data Types	Method(s)	Importance	Risk of Failure	Relevant QA program or responsible party	Include in DMS QA program?
	CTSim	General	Manual	High	High	Manual data entry — patient‐specific quality assurance process in place	‐
MOSAIQ	PRIMUS (SEQUENCER)	General, image, treatment parameters	Proprietary interface	High	Medium	Some informal testing following upgrades or repair, and some patient‐specific quality assurance in place	✓
PRIMUS (SEQUENCER)	MOSAIQ	Image, treatment parameters	Proprietary interface	High	Medium	Partially covered by linear accelerator quality assurance program, image quality not addressed by existing quality assurance	✓
PRIMUS (SEQUENCER)	Coherence therapist	General, image, treatment parameters	Proprietary interface	High	Medium	Some informal testing following upgrades or repair, and some patient‐specific quality assurance in place, image quality not addressed by existing quality assurance	✓
Coherence Therapist	PRIMUS (SEQUENCER)	General, treatment Parameters	Proprietary interface	High	Medium	Some informal testing following upgrades or repair, and some patient‐specific quality assurance in place	✓
PACS	Pinnacle	Image	DICOM	High	Medium	Pinnacle quality assurance program	‐
CTSim	Pinnacle	Image, general, treatment parameters	DICOM, DICOMRT	High	Low	CT‐Sim quality assurance program, and patient‐specific quality assurance	‐
Pinnacle	MOSAIQ	Image, Treatment parameters, documents	DICOM, DICOMRT, intranet	High	Low	Partially addressed by Pinnacle quality assurance program as well as patient‐specific pretreatment quality assurance	✓
MOSAIQ	SYNERGY (SEQUENCER)	General, image, treatment parameters	Proprietary interface	High	Low	Connectivity and data transfer tested as part of linac commissioning, some limited testing following any change	✓
SYNERGY (SEQUENCER)	MOSAIQ	Image, treatment parameters	Proprietary interface	High	Low	Connectivity and data transfer tested as part of linac commissioning, some limited testing following any change	✓
SYNERGY (SEQUENCER)	SYNERGY (LCS)	Treatment parameters	Proprietary interface	High	Low	Linear accelerator quality assurance program, and connectivity and data transfer tested as part of linac commissioning
SYNERGY (LCS)	SYNERGY (SEQUENCER)	Treatment parameters	Proprietary interface	High	Low	Linear accelerator quality assurance program, and connectivity and data transfer tested as part of linac commissioning
Coherence Therapist	PRIMUS (LINAC CONTROL)	Treatment parameters	Proprietary interface	High	Low	Linear accelerator quality assurance program, and connectivity and data transfer tested as part of linac commissioning
PRIMUS (LINAC CONTROL)	Coherence Therapist	Treatment parameters	Proprietary interface	High	Low	Linear accelerator quality assurance program, and connectivity and data transfer tested as part of linac commissioning
	Meditech	General	Manual	Medium	High	Manual data entry — process quality assurance needed
MOSAIQ Data Director	MOSAIQ	Image	DICOM	Medium	Medium	Data Director quality assurance program
Meditech	MOSAIQ	General (Demo)	HL7 interface	Medium	Low	Connectivity and data transfer tested at interface installation, repeated following any change (responsibility of hospital IT staff), data integrity in MOSAIQ not covered by existing quality assurance	✓
CTSim	Brachyvision	Image	DICOM	Medium	Low	Connectivity tested at installation of Brachyvision, patient‐specific quality assurance
SYNERGY (SEQUENCER)	XVI	General, image	Proprietary interface	Medium	Low	Linear accelerator and Onboard Imaging quality assurance programs
XVI	SYNERGY (SEQUENCER)	Image, treatment parameters	Proprietary interface	Medium	Low	Linear accelerator and Onboard Imaging quality assurance programs
SYNERGY (SEQUENCER)	iView	General, image	Proprietary interface	Medium	Low	Linear accelerator and Onboard Imaging quality assurance programs
iView	SYNERGY (SEQUENCER)	Image, treatment parameters	Proprietary interface	Medium	Low	Linear accelerator and Onboard Imaging quality assurance programs
Pinnacle	Coherence Oncologist	Image	DICOM	Medium	Low	Connectivity and data transfer tested at commissioning and following change, image quality not addressed by existing quality assurance	✓
Coherence Oncologist	Coherence Therapist	Image	DICOM	Medium	Low	Connectivity, image quality and data transfer tested at commissioning and following change
MOSAIQ	Orthovoltage	Treatment parameters	Proprietary interface	Medium	Low	Connectivity and data transfer tested as part of commissioning of new control software, not fully addressed by Orthovoltage quality assurance program	✓
Orthovoltage	MOSAIQ	Treatment parameters	Proprietary interface	Medium	Low	Connectivity and data transfer tested as part of commissioning of new control software, not fully addressed by Orthovoltage quality assurance program	✓
MOSAIQ	CTAR	General	Custom interface	Low	High	Connectivity and data transfer tested at interface installation, repeated following any change (responsibility of hospital IT staff)
MOSAIQ	Meditech	General (schedule)	Custom interface	Low	Medium	Connectivity and data transfer tested at interface installation, repeated following any change (responsibility of hospital IT staff)
MOSAIQ	MOSAIQ Data Director	Image	DICOM	Low	Medium	Not addressed by Data Director quality assurance Program	✓
Brachyvision	MOSAIQ	Documents	Intranet	Low	Low	Patient‐specific check

Table A4

Example Test Table for the DMS in Figs. A1, A2, A3, A4, A5, A6

Designator	Test example	Performance
Weekly
W1	Data integrity (MOSAIQ)	Complete
W2	Audit for data completeness (MOSAIQ)	Complete
W3	Monitor user and system logs	Complete

W1 = Patient data integrity in the MOSAIQ database is verified via the generation of custom reports focusing on known weaknesses in data transfer via interface (e.g., duplicate patients, scheduling errors, patients with alerts requiring specific measures or intervention, etc.). From designators L1 and C4 in Table 2 in this document.

W2 = Completeness of schedule status and workload data is verified via built‐in audit reports and/or custom reports within MOSAIQ. From designator C2 in Table 2 in this document.

W3 = Monitor user and system logs for unusual activity. From designator C2 in Table 2 in this document.