Artificial intelligence-assisted blockchain-based framework for smart and secure EMR management.

Healthcare professionals, patients, and other stakeholders have been storing medical prescriptions and other relevant reports electronically. These reports contain the personal information of the patients, which is sensitive data. Therefore, there exists a need to store these records in a decentralized model (using IPFS and Ethereum decentralized application) to provide data and identity protection. Many patients recurrently visit doctors and undergo treatments while receiving different prescriptions and reports. In case of an emergency, the doctors and attendants may need and benefit from the patients' medical history. However, they are unable to go through medical history and a wide range of previous reports and prescriptions due to time constraints. In this paper, we propose an AI-assisted blockchain-based framework in which the stored medical records (handwritten prescriptions, printed prescriptions, and printed reports) are stored and processed using various AI techniques like optical character recognition (OCR) to form a single patient medical history report. The report concisely presents only the crucial information for convenience and perusal and is stored securely over a decentralized blockchain network for later use.

Introduction

The healthcare industry is heavily data-driven and can make the most out of rapidly advancing science and technology. To some extent, doctors around the world are performing grassroots record-making using a pen, storing the data offline in huge stacks, and hence, crucial data are not readily available when required [1]. In order to meet the requirements in smart domains, 5G networks are anticipated to resolve the issues faced by the present 4G networks. This includes convoluted communication, the potentiality of device computations, etc. [2]. Blockchain technology has always shown its potential to keep persistent and transparent logs of data, making it a popular technology choice for protected applications [3-7]. Electronic medical record (EMR) system of storing sensitive patient data has been contemplated to hold a significant devoir in revamping the healthcare intelligence and improving the quality of end-user experience. It has been reported that EMR systems have the potency of saving more than billions of dollars a year [8], and using the latest technologies of 5G communication, artificial intelligence, and blockchain shall further improve the prospects [9-11]. Experts frequently observe that critical patients having diseases are intermittently indisposed to multifarious further conditions [12, 13]. In such scenarios, the patient could be incessantly visiting different specialized doctors. Since there may be a correlation between the diseases, they are not collectively short-lived and can take many years to cure [14, 15]. Several reports are advocating that a systematic history of the patient would administer the lion’s share of the data required for diagnosis [16-18]. This is supported by a study that concludes that more than 50% of physicians who responded considered patient history as the most decisive facet for diagnosis [19]. Patients having a long term disease usually visit various doctors during the course of their medical history. This comes with supplements of all kinds of documented prescriptions, laboratory reports, and other relevant data. Commonly these data are disorderly collected and stored, during which some hard copy of a document may get lost, whereas the others are stored without taking care of the timeline of the patient diagnosis history. In many instances where patients are not fully acquainted with medical literature of their complex cases, the patient may not be able to properly address their case to a new doctor, which leads to an ever-increasing impediment to the provision of good healthcare and treatment [20, 21]. Hence, the doctors at times are not able to recognize the patient’s history in detail because ambivalent and muddled information reaches them, which in turn leads to repetition of medicines and unintended negligence of some critical information. Such gross negligence may gradually lead to eventual avoidable fatalities. In case of emergencies, notably in outstation hospitals, it becomes strenuous and challenging for a new doctor to provide appropriate treatment to such patients. Generally, the doctors receiving patients with no proper previous medical records do not take such cases and refer them to other hospitals due to unclear diagnoses [22].

In order to share data independently among medical specialists, medical staff, patients, and other stakeholders, there is a call for a safeguarded infrastructure. EMRs are highly sensitive data, and as we store these data over the internet, higher is the vulnerability toward its unwanted disclosure. The healthcare systems in use today utilize a unified architecture demanding unified trust over a fast and reliable communication medium [23]. An efficient coalition of EMRs and interoperability among the currently present systems persists as an arduous exercise. Moreover, the patients possess limited to no regulation over their data [24]. To make the system completely automated and make it easy-to-use, AI-assisted techniques like OCR can be helpful. Because of the aforementioned shortcomings, there is a need for a system in which controls are user-centric, and the privacy of the users and the data are securely maintained.

Medicine, diagnosis, and treatment processes are ascertained to involve blockchain technology, which can exponentially upshot additional value [25]. Blockchain technology has emerged from bitcoin [26] and rests over pseudo-anonymity and public key infrastructure (PKI) by safekeeping the confidentiality of its end-user, i.e., the patient in the relevant scenario [27].

In this paper, we propose a streamlined platform using 5G, blockchain technology, AI, and machine learning techniques for both medical staff and patients to use, so as to benefit them by electronically illustrating the entire patient history in detail at once. This is done by creating a distributed network on the Ethereum platform.

Related works

Evans et al. [28] elaborated on the EMR system as an invention that aids and refines the active methodologies of sustaining and retrieving data. EMR system draws and maintains the patient information digitally, as compared to other non-digital systems, and can, therefore, gradually rule out or augment the presence of keeping data records physically in the healthcare system. A study marks the ability to document and examine the medical records electronically and how it is in the best interest of convenience for doctors [29]. Yet accomplishing quality breakthroughs with the use of EMR usually takes a toll on cost constraints and poses a challenge during the transition. A qualitative study of medical practices shows that the physicians who had enforced an EMR had their efficiency hinged heavily on how they use the EMR, rather than just implementing it [30]. The current use of EMRs hold some additional key disadvantages as identified in [30-32]. Machine learning- and deep learning-based models for image analysis have been proposed and studied in several studies [33, 34]. Chen and Shih [35] have implemented the architecture of the EMR system that can be conflated with streaming media. The static and dynamic patient media is encoded in a streaming video arrangement and is shelved in a flash media server (FMS). Along with this, the EMR records are transfigured to XML documents and harnessed for the integration of descriptions with embedded streaming videos. A system has been recently proposed by Xia et al. [36] that can effectively ad minister and protect medical records. It is a blockchain-based system that is equipped for smooth management and protection of shared medical data present in large data bodies. The data remain secure in the system by cryptographic keys and authentication. However, there remains a concern about the vulnerability of the system from data disclosure [37]. The medical institutions are usually unwilling to provide data to a third party, thereby making the system flawed. Esposito et al. [38] have stated the shortcomings of adopting a digital methodology of storage on the cloud in instituting a data-sharing platform in a healthcare system. The potential obstacles, including privacy protection, of using blockchain technology, have also been addressed in [38]. However, it does not advocate any feasible solutions to counter the challenges addressed.

Guo et al. [39] detailed a scheme that acclaims the legitimacy of electronic health records (EHRs) stored in a blockchain; the attribute-based signature scheme, along with numerous authorities, is architected for the same. Post-completion of the treatment of the patient, all the patient data along with insurance documents, EHRs are stored in a single block of blockchain. Usually, the size of data stored in a single block is small to maintain security standards; however, medical data such as imaging and other arrangements can be large in size, relational, and may have the desire to search. DRAMS has been introduced by Ferdous et al. [40] having a decentralized blockchain cognition scheme to ensure a control system having distributed access. The infrastructure possesses an aid to secure data; however, a lack of efficiency in data sharing remains a problem.

A blockchain AI model was proposed by Wehbe et al. [41] outlining the requirements for a computer-aided diagnosis system using healthcare records. The authors of [42] used a private blockchain network for AI-enabled drones to provide various healthcare services. Zhaofeng et al. [43] proposed a blockchain-based framework for data management systems in edge computing. Jabarulla and Lee [44] propose a blockchain and AI-based framework for handling the COVID-19 pandemic-related issues.

Although there have been many works on EMRs and multiple models have been developed from them, the problem persists due to the shortcomings in the models leading to their limited use. The major contributions of our work are as follows:

We are proposing a system that is inspired by the aforementioned works and not only integrates the features from each of them but also adds additional features, which are the need of the hour. Additionally, the framework we propose can be integrated with 5G technology seamlessly for a fast and reliable system [45].

To the best of our knowledge, our prototype would be the first one to have EMRs stored on the blockchain, will be completely decentralized, securely store patient data, will generate a summary of patient history using technologies like OCR, and will be easily accessible by doctors, patients, attendants and all other concerned and authorized stakeholders.

We attempt to add more value to the EMR system so that the doctors can use it efficiently and frequently, increasing the accomplishing quality of medical professionals, as mentioned in [29].

The proposed framework

Figure 1 illustrates the steps involved in the proposed system model. When a patient joins the platform, they submit pictures of all their prescriptions and laboratory reports. The Ethereum platform securely stores all the media on a blockchain ledger and feeds the data as per request into the machine learning model.

Fig. 1

Overview of the system model and effects on the stakeholders of the framework

There have been various works in recent years that use different machine learning concepts to improve healthcare applications [46-49]. The data submitted by each patient is prepared, refined, and processed altogether by the system of intelligent processes, and a single summary is generated for each patient. This summary provides the medical history of patients in a quick and illustrative manner and can be accessed and updated by doctors, attendants, and patients. The steps involved in the execution of the aforementioned framework are as follows:

A distributed network on the Ethereum platform is constituted. The doctors, attendants, patients, and potentially any healthcare professional or stakeholder can join the network.

All members enter the network using a secure login interface. If a new patient is joining the network over 5G, all they have to do is upload pictures of available prescriptions and medical reports onto the network. In the case of an existing member patient, they have to update the system by adding new prescriptions provided to them.

All the media are essentially uploaded to IPFS Image Upload dApp (decentralized application) with Ethereum smart contracts, while the permanent and immutable IPFS links (hash) are placed in the blockchain transaction. This is done because storing heavy data like images requires large amounts of gas. However, IPFS hashing keeps up the integrity of the application by upkeeping the security.

Hash values are stored in Ethereum blockchain as per the type of document. For summary reports, the ledger is scanned to obtain the new IPFS hash links in python. These are referenced to the media resting in dApp.

As explained in Fig. 2, the model reads the reports in different layers; firstly, the handwritten prescription images are preprocessed to improve quality, by preparing it best for the algorithm applied. The documents are then processed using various machine learning algorithms based on AI to extract words, which in turn are recognized using optical character recognition (OCR), and ASCII values are stored.

Fig. 2

Network model for the proposed framework

The printed prescriptions and other printed reports are processed by Microsoft Azure Computer Vision to obtain relevant information from the documents and can be stored on any encrypted storage NoSQL database platform.

The basic information of the patient is obtained, i.e., name, age, sex, weight, blood pressure, symptoms, existing conditions, medications (whether any of them is life-critical/SOS), allergies, etc. All the data that are extracted from multiple files are collectively stored as a single document. It is ensured that during all times, whether the information is resting or on the go, the data remain secure.

The doctors and patients can view this summary of patient history conveniently. Whenever the patient visits another doctor or submits another report or prescription, the summary report is updated. A user-friendly front-end platform is deployed for healthcare professionals that pictorially shows them the entire case in a concise manner. This would help the doctor to easily and quickly go through the patient history. This would also help and prevent the patients and attendants from the hassle of managing, storing, and showcasing all the required documents as and when required. The patient can visit the doctor just with his ID, and the doctor can see all the related history on the portal itself. All the communication and data transfer happen over a fast network of 5G.

In the next section, we discuss the network model of the paper in terms of the proposed utilities and the step-by-step implementation of the proposed work.

The model architecture

The network model can be broadly divided into four components, the first being a web front-end which receives and serves the content, the second being the storage of files in IPFS, the third being blockchain storage back-end, and the fourth being a comprehensive machine learning model processing back-end to generate results. Figure 2 summarizes the proposed architecture of the network model. We further discuss the implementation of each of these components of the network model [50]. Table 1 gives the details of the mathematical notations used in the model.

Table 1

Mathematical notations and their meaning

S. No	Symbol/Notation	Meaning
1	H	Data header volume of the blockchain
2	Gσ	Gaussian function kernels
3	σs	Filtering amount
4	I	Intensity value
5	ζ	Offset parameter
6	E	Expected value
7	µ	Mean intensity value
8	rCR	Character recognition accuracy

Front end

A compelling user interface (UI) allows the new user to securely register to the network either as a medical professional or as a patient. The credentials can be authenticated via a secure distributed or centralized system, providing users with a unique ID from which they can log in at any time. This top-level service is responsible for handling uploads and submits of images/pdf scans from the user. The users are prompted to upload the image at the right angle and not upload a rotated image. The users are also encouraged to scan the documents from a mobile scanner for better results. Since it has to feed the media into three blockchains as per the image type, the patient-user adds tags to each image during each upload. Broadly there are three types of tags: PrintedPrescription, HandwrittenPrescription, and PrintedMedicalReport. Additionally, the patients can view their generated medical report summary, and authorized medical professionals can view reports of the patients by searching via IDs and may refer the patient to another appropriate doctor if required.

Blockchain and IPFS

Each file is uploaded with a tag linked to it, and as per each tag, the system takes the file to IPFS to link it to the corresponding blockchain.

To build the project, a directory is created in which the React Truffle box is used. Truffle is a framework that allows us to rapidly build decentralized applications (dApps) on Ethereum. Truffle box automatically deploys a smart contract called SimpleStorage.sol, which shall be updated as per the requirement. This smart contract is used as a basis for storing the IPFS reference (hash) on the blockchain. To connect the smart contract and Truffle project to the blockchain, the network configuration file of the Truffle project is updated with the same port as used by Ganache to run the blockchain. For development, Ganache and Infura are used. Ganache is a simple tool to run a private blockchain on a local machine. It aids a blockchain to connect whenever we want to store IPFS hash on the blockchain and provides a way to deploy smart contracts. A connection is made to an IPFS node instance called Infura, which is a decentralized infrastructure that hosts Ethereum and IPFS nodes. It uses RPC protocol to connect to any remote Ethereum and IPFS nodes, without having the user run one manually.

Next, the file is converted into a stream of binary data that is comprehensible to perform the transfer task. This is achieved by using a buffer that allows sending the image to IPFS. Upon submission and successful upload, an IPFS hash is returned.

The web layer is now wired to not only upload the image to the IPFS but to write the received IPFS hash to the blockchain.

When a request is made for creating a transaction on the blockchain, Metamask requests the user to pay for the same. Transactions writing to Ethereum cost gas, and the transaction cost (Tcost) is calculated as per the following formula:where gasLimit and gasPrice values are decided and regulated by miners, and they change for every block. Storing IPFS hash on the blockchain is less expensive as compared to storing the images directly. The following equation represents the storage amount required for storing the hash values (S):where HashSize is the size of IPFS hash for each transaction, N is the number of transactions or hashes stored in the blockchain, and T is the native size of each transaction. When the hash is returned via IPFS, it is also stored in a table with a patient ID. Algorithm 1 explains the procedure of document upload to IPFS and storage on the blockchain.

Processing of files

The hash values corresponding to handwritten prescriptions, printed prescriptions, and reports are loaded onto their respective blockchains.

To make the AI-based machine learning code interact with the blockchain, Web3.py library is used. After pulling Web3 and JSON into the python file, the Infura (Ganache) URL is used to con nect to Web3. For establishing a connection with Ethereum, a smart contract is called. It’s required to have abi, which is a JSON array that describes how the smart contract looks like, address, which is the address of smart contract deployed on the blockchain, and private key of the blockchain account. These variables reconstruct the smart contract in the python environment and aids in obtaining IPFS hash values stored in the blockchain. Data extraction is performed by periodically calling the python function to run on the newly added hash values on respective blockchains.

The extracted IPFS hash values are first compared to the values stored in the table with tags and sent to the corresponding sub-utility. As discussed above, there are three types of documents uploaded on IPFS, and their hash values are, respectively, stored to three blockchains as per the tag assigned to it, namely

Type 1—Printed prescription

Type 2—Printed medical report

Type 3—Handwritten prescription

To extract data, the IPFS hash, along with the corresponding type of the document tag, is fed to respective sub-utilities. Now, we discuss the working of each of the sub-utilities in detail.

Sub-Utility 1

The IPFS python library is installed, and the connection is established with IPFS Peer via Infura. Thereafter, the document is obtained from IPFS using the IPFS hash as an input. In order to obtain the best results, the image is preprocessed in the following ways:

Rescaling: Tesseract provides the best results with images having at least 300 DPI. A check is placed to ensure whether the size of the image is greater than 300 DPI. If that is not the case, then the image is resized appropriately.

Removing shadows: In order to improve results, the image is run through the RGB split planes, then appropriately dilated, normalized, and appended back and then merged together.

Noise reduction: To eliminate noise and retain image edge structure, a nonlinear adaptive bilateral filter is used. The bilateral filter is deliberated by Tomasi and Manduchi [51], and the updated value of pixel in regard is calculated by using Gaussian distribution weights of neighboring pixels. The new image (I∗) is calculated as:where S is the window, Gs , Gr are the Gaussian function kernels, σ, σ are filtering amount, and W is the normalization factor given as:An enhanced variant of Eq. (3) was suggested by Zhang and Allebach [52] by introducing a novel offset parameter ζ, revising Eq. (3) to become:For improved performance to have sharper and smoother images, ζ and σ parameters are made to be locally adaptive.

Binarization: Sometimes image background can be of uneven colors leading to suboptimal results during thresholding in Tesseract. An additional layer of locally adaptive binarization is involved to transform the image to bi-level format and improve the results. The image is first converted to grayscale, and then, the threshold value T (x, y) at location of a pixel (x, y) present in the image is calculated as:T (x, y) is calculated by subtracting weighted average from a fixed constant parameter δ. The weighted average of region of interest is calculated using OpenCV, which provides two ways for doing the same, i.e., mean weighted average and Gaussian-weighted average. For the dataset used, we observed that Gaussian-weighted average thresholding gives the best results.

After going through the aforementioned preprocessing, the image is sent as an input to the AI-based Tesseract OCR Engine using pytesseract library on python, which is Google’s open-source Tesseract OCR Engine. Apart from the previous binarization, Tesseract uses Otsu’s thresholding [53] to improve the results. The image is first divided into m × n smaller images. If any small image doesn’t have text, then results of Otsu’s method will not be as accurate. Hence, the text property of small images must be calculated before further processing. The variance of ith smaller image σ2 is given as:Here, mean intensity values of ith smaller image are µ. It is suggested in [54] that if then the ith smaller image is non-text and all its pixels will be set to 0. The remaining smaller images are transferred to the Otsu’s method.The pixel value is stored in (xi, yj), (0, 0) is standardized as the first pixel of the image and (M−1, N−1) is the last pixel pair present in the image. While every pixel contains its RGB value, the pixels having equal RGB variable will come under gray color. With this, the formula to convert colored image to black and white is formed as follows:where r, g, and b are, respectively, red, green, and blue color values of the given (x, y). Mean value (MU) is always between 0 and 255, being given to green, blue, and red layers of (x, y) as represented by equations below:The steps mentioned above convert the image into grayscale. However, the process has been additionally supported by preprocessing.

Pytesseract output is the textual data of printed medical prescriptions that is temporarily stored in a variable.

Sub-Utility 2

Similar to Sub-Utility 1, the images are obtained from IPFS. In medical reports, the abnormal values and corresponding test names are usually in bold format. python-docx package is used to iterate through the file, and in case a bold value is detected, the entire line is stored in a dictionary variable.

Sub-Utility 3

Similar to Sub-Utilities 1 and 2, the image from IPFS is obtained. This utility is made for extracting data from handwritten prescriptions. Microsoft Azure Computer Vision Cognitive Service API, which is an AI-based platform, is used for the same. This API calls a RESTful Azure Service by uploading the retrieved image from IPFS and invokes the service to perform OCR analysis on the image. Computer Vision API returns a JSON response holding the OCR results. Before performing image feed operation to Computer Vision API, the following checks are made to ensure the image supports with the API requirements:

Image must be greater than 50 × 50

Image size is less than 4 MB

Image extension must be GIF, BMP, PNG, or JPG

In case the image does not fulfill the requirements, then appropriate actions are taken to fulfill the requirements.

A JSON response is procured from the API endpoint, which contains an array of tags. Each tag has the text and a confidence percentage linked to it. This information is also stored in a temporary variable similar to Sub-Utilities 1 and 2. Utility 1 passes all information ahead in python code to Utility 2. Utility 2 first checks in the centralized storage for the summary report of the patient. If it does not exist, a new file is created into which the variables are fed. All variables storing extracted data for the patient are appropriately scanned, and data are placed on the report summary, which is then sent to the centralized server for storage. If there exists a summary report in the server, then it is extracted, and the new variables are used to update the report, and thereafter, the updated summary report is sent back to the server. There are many measures available to calculate OCR performance. Common metrics available like r, recall and precision are given as:where n is the count of correctly identified characters, m is the count of incorrectly identified characters, and all is the total number of characters in the image. However, Eq. (13) holds some limitations, as it provides a negative value of rCR when m is greater than n. Therefore, a revised equation suggested by [55] is used:where all variables hold the same meaning as mentioned in Eqs. (13, 14, 15).

Results

Storing large amounts of data on the blockchain is expensive, and to minimize the costs, IPFS hashes are stored on the blockchain. Figure 3 shows the projected difference in cost occurrences as the number of transactions increases on the blockchain. It mainly projects the costs of writing data on the blockchain, as it is the more expensive transaction than read from the blockchain. As shown in Fig. 3, there is a drastic decrease in the costs of storage when the IPFS hash is stored instead of the entire image.

Fig. 3

Projected storage costs of images v/s IPFS Hash on blockchain

The dApp stores media on IPFS, which in turn stores the media with the aid of peers. It is shown in Fig. 4 that as the number of peers accessing transaction increases, or as more number of peers contribute to the storage of an image, the execution time or time required to procure the media increases. Additionally, it is also seen that with an increase in the size of the image or report, the time taken to access the same increases.

Fig. 4

Execution time v/s document size according to number of peers

Along with 5G network integration for overall fast and efficient communication within the framework and generation of responses, for the purpose of handwriting recognition OCR in Type 2 documents, multiple AI-based cloud platforms were taken into consideration. All the platforms can be procured via the APIs, and since they all have cutting edge performance, the relevance of choosing the right platform for our model has to be in accordance with the costs. This is because the platform shall support patients, doctors, and other medical staff at free of cost or minimal costs. As observed in Fig. 5, among AWS Textract, Google Cloud Vision, and Microsoft Azure Computer Vision APIs, the overall projected costs till roughly 5 million transactions of OCR came out to be least for Microsoft Azure Computer Vision API. Therefore, it was used for the required purpose in the model.

Fig. 5

Projected costs for cloud platforms with increasing API calls

Table 2 summarizes the precision, recall, and character recognition accuracy for Type 1, Type 2, and Type 3 documents. It is observed that the model shows the best accuracy for the Type 1 document. It is also important to note that the model is always subject to improvement.

Table 2

Performance measures of report types

Document type	Precision (%)	Recall (%)	Character recognition accuracy (%)
PrintedPrescription	83.047	86.044	84.211
HandwrittenPrescription	71.332	74.218	72.001
PrintedReport	89.637	80.119	89.333

A number of preprocessing filters are applied to Type 1 and Type 3 documents before they are fed to the AI-based Tesseract OCR for processing. Figure 6 depicts the character recognition accuracy before and after preprocessing. Preprocessing the images results in an overall improvement of 4.698% in the performance of the model. Another notable observation is the net information viewed by healthcare professionals. Since there is a time crunch for doctors, they require patient information in a concise and visually sound manner. As the doctors view only the summary reports generated by the model and not all the documents, Fig. 7 shows a decline in increasing data of patients; large amounts of data are avoided as the number of patients increases, and only important information is preserved for doctors to view, thereby reducing the time taken by healthcare professionals for analysis.

Fig. 6

Tesseract performance with v/s without preprocessing

Fig. 7

Total storage space acquired by documents in total v/s summary reports

Conclusion

In this paper, the outline and execution of synergized Ethereum blockchain and IPFS decentralized application (dApp) over 5G network are presented for the upkeep and storage of patient documents. Along with the use of AI-based machine learning techniques and data analysis, a concise summary report is prepared for perusal and examination by healthcare professionals and patients. For the purpose of maintaining minimal costs and scalability, IPFS hash values are stored on the blockchain instead of whole documents. In addition to the storage of documents, Computer Vision API calls, along with other techniques, are used to prepare summary reports. The model is optimally decentralized and therefore maintains the required security standards without compromising on patient identity, thereby furnishing upright and convenient assistance to the authorized peers.