Selection of computational environments for PSP processing on scientific gateways.

Science Gateways have been widely accepted as an important tool in academic research, due to their flexibility, simple use and extension. However, such systems may yield performance traps that delay work progress and cause waste of resources or generation of poor scientific results. This paper addresses an investigation on some of the failures in a Galaxy system and analyses of their impacts. The use case is based on protein structure prediction experiments performed. A novel science gateway component is proposed towards the definition of the relation between general parameters and capacity of machines. The machine-learning strategies used appoint the best machine setup in a heterogeneous environment and the results show a complete overview of Galaxy, a diverse platform organization, and the workload behavior. A Support Vector Regression (SVR) model generated and based on a historic data-set provided an excellent learning module and proved a varied platform configuration is valuable as infrastructure in a science gateway. The results revealed the advantages of investing in local cluster infrastructures as a base for scientific experiments.

Introduction

Among the many computational systems available for scientific analyses, in silico experiments offer advanced results and demand higher computational capacity. Non-specialized users have shown difficulty in using complex computational solutions, due to their particularities. Therefore, simplicity of use must be one of the fundamental characteristics of such systems, once transactions and computations must be transparent for facilitating access and manipulation of the available resources.

Science Gateways consist of a set of tools, applications, and data integrated via a user-friendly portal and their main objective is to enable a larger group of users to conduct experiments, even if they are not skilled for dealing with the details of a computational resource configuration. Typically, they use tools as workflow management systems, e.g., Galaxy [1], Taverna [2], gUse [3] among others [4], [5], [6], [7], [8], for reproducibility and simplification of execution processes [9]. Such systems are integrated with databases for the acquisition of workflow steps and input data or storage of the processing results [10]. On the other hand, Science Gateways may suffer from performance traps. As they offer a high-level solution, users do not have information about servers location or installed capacity. Systems for sophisticated computations are expected to offer enough capacity, regardless of the complexity of the processing parameters, which is not always true. Hardware limitations can hamper the experiments performance, slowing the progress or delivering poor results. Protein Structure Prediction (PSP) tools, designed for discovering the native structure of a protein based on its amino acid chain [11], are an example of such complex experiments, once they demand strict controls of computer performance and generate intensive data.

Among the Science Gateways challenges are (i) selection of a gateway that is actively maintained, (ii) discovery of new services, (iii) real-time service monitoring and management, (iv) identification of sufficient computing resources for the problem complexity, and (v) support from the user's community [12]. Science Gateways must deal with deadline constraints and paid resources for processing experiments in constant growth, therefore, the allocation of more computational resources from cloud computing is a reasonable solution. Limitations on the allocation of heterogeneous systems include different cloud providers that are not inter-operable [13] and nonexistence of a module, framework or strategy that guides the user towards the definition of an optimized computational resources configuration for running experiments.

This paper proposes a strategy based on data mining techniques for the understanding of the relationship between users input data and the behavior of the system over the execution time. The results showed Scientific Gateways require a module to help the decision on the type of resources allocation for the execution of scientific experiments. The contributions of the approach can be summarized into:

an extensive evaluation of Galaxy capacities and shortcomings;

a comparison of processing PSP in a varied set of machines; and

a base module for any Science Gateway to support the computational resources identification for running experiments.

The remainder of the paper is organized as follows: Sections 2.1 and 2.2 briefly review the Service Oriented Architecture and the concepts of protein prediction structures, respectively, Section 3 introduces Galaxy framework and its mechanisms; the main proposal, materials and algorithms are presented in Section 4; Section 5 addresses the benchmark setup and the performance evaluation of executions in Galaxy environment; Section 6 reports the main works related to Science Gateways; finally, Section 7 provides the concluding remarks and suggests some future work.

Background

This section presents the background for this paper, which includes SOA and PSP. In both subsections are included some references of each research field.

Service oriented architecture

Research institutions routinely deal with various types of data and coding systems, which are fragmented and spread among data repositories and lead to information loss and increased costs of their systems [14]. Cross-platform integration can be provided through a Service-oriented architecture (SOA)-based solutions that support scalability, re-usability, and heterogeneity [15]. Therefore, web services integrate different repositories and platforms while maintaining their privacy policies and adaptability [14].

A SOA-based system is comprised of the following three main elements: (i) service provider, which hosts independent web services to perform procedures; (ii) service repository, which stores the description of services, location and accessibility; and (iii) client application, which requests services to a host based on its service description [16].

Services can be integrated and distributed by public clouds, dedicated clusters or multi-architecture systems with processing GPUs. Besides those dynamic environments delivering high performance computing the requirements for a dependable architecture involve heterogeneous resources, scalability and minimization of communication, among others [17], [18], [19].

Abdul-Wahid et al. [20] proposed a framework based on Work Queue (WQ) and python implementation that provides scalability and integrates clusters with cloud solutions for concurrent processing. Pronk et al. [21] worked with three levels of parallelism (SIMD, threads, and message-passing) and provided automatic resource allocation. Such studies show the complexity of integrating resources and technology for research and the ways computation can act.

Protein prediction structures

This section describes the protein chains and the PSP processes for a better understanding of their importance and use as workload. Proteins are linear polymer chains that perform fundamental biological functions for the maintenance of life [22], once they regulate most activities within live organisms, as replication of genetic code and maintenance of cell shape [23]. Proteins are formed from amino-acid sequences (also known as residues) linked by peptide bonds. Two amino acids form a peptide bond when they concatenate by releasing water. The folding process aims at defining the native state of a protein from an inactive chain [23]. Scientists have focused mainly on two techniques to define those structures, namely X-ray crystallography and nuclear magnetic resonance (NMR). Although precise, the technologies are costly and the processes are slow, which have motivated the research on PSP, which aims at predicting the tertiary structure of a protein based on the amino-acids chain.

The folding process is a physical-chemical method in which any of the strands displays a functional configuration for a specific 3D structure [24]. Four basic representations of proteins are defined according to the residues arrange: the primary structure is the representation of the amino-acid chain; the secondary represents the hydrogen bonds and a basic order; the tertiary structure is the three-dimensional (or native) form, which enables the protein to perform biological functions. It is energetically stable and can be used in drug design or for the understanding of behavior of diseases. Finally, the quaternary structure is a representation of multi-tertiary complexes. Figure 1 shows the four structures [24].

Figure 1

Four types of protein structures [24].

The amino-acid chain is informed as input for the PSP processing phase. In the sequence of the execution, many structures are produced until the best one has been achieved, which represents the most stable one. The identification is defined by the structure of lowest energy and can be expressed by the following formula: if where represents the family of valid sequences for a given set and is the energy function of the desired structure, the general formula is defined as [25].

Research on prediction of protein structures has faced obstacles, despite the evolution of technology [25]. The main objective of researchers is to improve the prediction quality, however, they rarely focus on computational efficiency or reduction of processing time [26].

The impact of similar research on public health is an example of the importance of the field. Over a decade ago, Amato et al. [27] revealed scientists resorted to distributed systems, which are currently much more advanced. Other approaches are presented in [28], [29] and [30]. This field is close to health-related research, as in [31], [32], [33] and [34]. Reduced prediction error is presented in [35], and improvements in information on protein structures can be found in [36], [37] and [38]. Finally [39] describes the I-Tasser framework hosted on servers exclusively for PSP. The majority of improvements are protein-related methods.

Many of the above-mentioned studies suffer from performance and computational provision problems. Cloud Computing is the preferred platform, once it is powerful; however, it implies cost limitations and third-part management. The integration of clusters and other platforms can be an important advance for ensuring higher computational capacity and reductions in the processing time. Some of the studies presented must be operated by scientists with little experience in complex systems. As Science Gateways are systems with tools for processing and data analysis while maintaining transparency, they simplify the access to users. As the PSP experiments are computationally consuming, they require a careful infrastructure definition and help for the identification of limitations of those gateways.

Materials

Galaxy is an open-source and modifiable web platform widely used in research on bioinformatics. Servers are available for free and users access analysis tools and mechanisms for running and reproducing workflows. Some algorithms, methods and analysis tools can be integrated in the framework, which makes it personalized and adequate to users' needs [1]. The main goals of Galaxy are (i) access to complex computational resources from a broad public, (ii) reproducibility of experiments and (iii) collaborative analyses via web [40].

Galaxy has been built for an easy access by unskilled users. It has a general purpose and is used in several domains with the same quality. Some of its principal features include accessibility, reproducibility and transparency. The Galaxy Project is divided into the following parts:

Galaxy Server: The Galaxy project1 provides computational resources to Bioinformatic experiments. Public servers can be found in other research pages.

Galaxy software framework: an open source project that offers customization and integration to provide services. This research has been conducted in an instance developed at the Institute of Mathematical Sciences and Computation (ICMC)2 and named Koala,3 which provides data management, PSP predictors, analysis tools and workflow assistance. The interface is shown in Figure 2 [41].

Figure 2

Example of a Galaxy implementation available for use at ICMC servers.

Galaxy Tool Shed: space for the sharing of tools, solutions and steps for configuration and installation. It is available at the project page.4

Galaxy Community: the main part of the project, it is composed of developers, users and administrators that work together for evolution and updates. Discussions and collaboration are primordial for any open-source endeavor [1].

In comparison to other platforms, Galaxy stands out because of its many resources and great community [1]. It can be used via web interface, regardless of its installation and is a considerably easy system for manipulation. Figure 3 shows the process of use of Galaxy Koala architecture. The user inputs the data, usually collected from a public storage, as PDB. They are informed to the framework via direct download or upload tools. The user chooses the algorithms that process the data according to the experiments design. If necessary, intermediate data (i.e. population files of a genetic algorithm) are produced. The processing phase starts, the results are generated, and the user decides on storing or moving the data.

Figure 3

Sequence diagram of an experiment in Galaxy Koala.

Despite its penetration in scientific fields and its many tools that help research conduction, several aspects of Galaxy (and also Koala) have not been optimized. Some important limitations regard monitoring of active processes, difficulties with large data-sets and lack of information about machine hosts. The lack of information on computational capacities limits the design of experiments and can cause failures that cannot be avoided by users. The system cannot detect or inform on the problem, and in an attempt to finish an execution, it can cause a dead lock. The user receives zero notification about the system or the experiment status and waits for a conclusion that may not be reached. The access control is also a problem, once only the administrator can check and fix most faults. Galaxy has been designed for a broad public, including non-skilled users, which is a major drawback in its usability. Once Galaxy Koala' hosts cannot be chosen, the user must rely on the instances offered.

A survey into Galaxy Koala components, types of faults and extent of damage based our previous studies and the literature review is shown in Table 1. The components may be tools offered by Galaxy that report mechanisms or hardware devices. Predictors are tools developed for PSP experiments, providers nodes are the hosts that offer the services and feedback is the mechanism that informs users on the conclusion of the experiment. Detectability describes the system operation and frequency indicates how often faults occur. Such shortcomings indicate the improvements required and that the maintenance of Science Gateways must be constant. Users' needs, technologies available, complexity involved and possible gains must be accordingly evaluated. Problems similar to the ones in Koala are found in other Galaxy implementations [1].

Table 1

Galaxy Koala components and most common failures.

Component	Failure	Domain	Consequences	Detectability	Frequency
Predictor	Crash	Content	Medium	Yes	Occasionally
Predictor	Input	Content	Minor	Yes	Often
Memory	Insufficiency	Time	Major	Yes	Often
HD	Insufficiency	Content	Fatal	No	Rare
Feedback	Crash	Time	Minor	No	Often
Monitoring	Nonexistence	Content/Time	Major	No	Often
Provider Node	Crash	Content	Major	Yes	Occasionally

Methodology

Critical points of Galaxy Koala were identified in our assessments. The modules common to Galaxy and other Science Gateways can be defined after a careful study, as shown in Figure 4. The portals are accessible through an interface, usually a web page, for authentication and definition of access levels. Workflow managers define the sequence of tasks execution, as well as data management. Management tools, such as schedulers, monitors, and load balance comprise the next module. Finally, the execution module forwards the requests to machines and repositories for execution, storage and retrieval of information. The extra module in this infrastructure is the Decision Maker proposed here. This component performs a previous analysis of the user's needs and indicates the computational configuration for the execution of an experiment with best cost-benefit. It differentiates the best environment for a request and reduces costs of cloud machines and network traffic.

Figure 4

Basic common modules of procedural architectures. The Decision Maker module is a gap in this technology that can help the creation of scientific processes with good performance.

Decision Maker uses machine learning to process past runs, discover patterns on a system's behavior and define the best configuration for each data entry. It uses this information to toggle the sending of requests between infrastructures installed locally in desktop machines, computer clusters, or cloud instances. According to the parameters, it chooses between running on local servers with downloaded information or sending the data to remote servers.

Environment selection

On many occasions, defining the configuration for a processing environment towards obtaining the best performance is a hard task. A machine of high capacity can probably process many experiments, but also wastes electric power and computational resources when dealing with smaller tasks. Modest resources, on the other hand, may cause delays, wrong answers or process starvation. Another problem is related to inflexibility in classic workstations, limited by hosts' capacity and difficult update. Clusters combine the capacity of various machines and are a good option to offer high-demand resources. They are also limited by the number of machines available and create parallel computing complexity to programmers. The other option is cloud computing, which offers elasticity, pay-as-you-go models and availability in a range of prices and according to the investment. If researchers have to deal with all such options during experiments, the quality is reduced and errors can occur.

Decision Maker makes decisions over the configuration of the machines that will execute the experiments. The users' parameters combined with previous experiments are used in evaluations for the definition of the best solution to each case. The outputs of this module are the best suitable configuration of an environment such as a workstation, a VM on cluster or cloud instance. This study aims at contributing to research on PSP providing an environment in which experiments yield results in an optimized way, by either cheaper and/or faster methods. Therefore, Quality of Service (QoS) metrics, usual in SOA, must be respected and specific definitions for the research field must be provided. This article proposes an infrastructure based on multi-environments, selection of the best configuration for each set of experiments and supply of resources in an integrated form. Galaxy Koala and its tools can be offered through a diverse set of machines that work either locally or via the Internet. The following three setups were defined and classified by the computational capacity: a desktop machine, a cluster machine and a cloud machine. The execution of a given experiment in the proper machine can avoid delays and waste of resources. Figure 5 shows the heterogeneous environment in a crescent order.

Figure 5

Environments considered for the architecture.

Computational capacity

Computational resources may vary regarding capacity. The performance of desktops has significantly improved, however, the two main resources for scientific purposes are clusters and cloud computing. Each of them includes advantages and shortcomings for maintaining Service Level Agreements (SLA) [16]. Galaxy Koala can be installed in any of such resources, and the choice will vary according to the initial specification and experimental design. When running in a private cluster, the administrator can choose among many aspects for configuring an environment and deciding on the steps of an experiment. Working locally is a good choice for researchers who have expensive computers at hand and are technically skilled to manage the system.

Clusters are a congregation of machines prepared for parallel computing and to offer combined capacity. They are usually available in companies and universities and some of them offer public access. They are a very good option for running experiments, and, in most cases, although the final user does not need to deal with the maintenance of hardware and software, the administrator's privileges are lost.

Finally, cloud computing provides resources on demand. It could be the solution to every capacity problem, however, costs and bandwidth are some of its limitations. The Internet speed has not grown proportionally to the size of data generated, which represents a prohibitive aspect in many cases. When running experiments exclusively on the cloud the ideal is to allocate virtual machines closer to the database, for the avoidance of data transportation and storage the results in disperse repositories. Such aspects brings new concerns regarding the maintenance and privacy of information.

To work as the computational environment, a diverse set was defined as follows: the working machines were divided into three different classes, namely “desktop”, “cluster” and “cloud” and their different configurations are shown in Table 2.

Table 2

Three Galaxy instances.

Environment	Hard Drive	CPU	RAM	Operational S.
Desktop	20 GB	2 Core	8 GB	Linux Ubuntu 14.04.4 LTS
Cluster Node	50 GB	4 Core	16 GB
Cloud VM	200 Gb	8 Core	32 GB

Algorithms

This section introduces the machine learning algorithms used for learning from data obtained by previous protein experiments in the Galaxy Koala platform. Each data point is referred to as vector , in which is a d-dimensional vector of input variables (or features) and y is a single output variable, i.e., the target variable. The objective of a learning algorithm is to produce a predictor , trained by the learning set . If the produced is a good predictor, i.e., if it has learned from data, it can predict unseen data with a small error, as long as this data point has been sampled from the same distribution of .

As the target variable of our experiments is a numerical value (the processing time), algorithms referred to as prediction modeling were chosen. Three different algorithms of linear regression, and one based on Support Vector Machines (SVMs) were assessed. Different algorithms were chosen for the task because different machine learning algorithms learn differently from data, i.e., depending on the data, some learning algorithms are better to learn than others.

The linear regression method approximates a formula as the one shown in Equation (1), in which is the vector of coefficients. Therefore, the obtained predictor is a dot product between input variables and coefficients W estimated by the minimization of squared error, shown in Equation (2).

Smaller coefficients W are desirable, because they reduce overfitting [42]. The two approaches for linear regression that are more effective for such reductions are ridge regression and Lasso. In Ridge, an -penalty term is summed to objective function O of Equation (2). Such as a , where , penalizes the linear regressions with larger coefficients. On the other hand, Lasso algorithm uses an -penalty summed to O, , and obtains sparse solutions, i.e., solutions with few nonzero components. Such a characteristic is specially effective when irrelevant features are present.

SVMs were initially developed for the binary classification of numeric data. They focus on the optimization of hyper-plane that maximizes the margin between the two classes, which is proven to generalize well to unseen data as it is grounded in the framework of statistical learning theory [42].

An SVM can be used as regression using the method ϵ-SV regression [43]. Its goal is to find a linear function , with , and , that has at most ϵ deviation from the actually obtained targets for all training data, i.e., if the prediction error is lower than ϵ, it is not considered. At the same time, it seeks small w, a property called flatness. The advantage of the use of SVMs is they can be coupled to kernels, thus transforming the input data into another, higher dimensional space, which enables the separation of nonlinear data.

Results and discussion

An elaborated scenario was prepared on the Galaxy Koala Framework for the collection of results and modeling of data behavior through machine learning, so that the best computational environment could be properly defined. Experiments were performed at cluster Andromeda available at the Distributed Systems and Concurrent Programming Laboratory (LaSDPC)5 (see details in Table 3).

Table 3

Andromeda cluster details.

Cluster Andromeda
Number of Hosts	13
Processor	AMD Processor Vishera 4.2 Ghz
RAM	32 GB RAM DDR3 Corsair Vegeance
Hard Drives	HD 2TB Seagate Sata III 7200RPM
GPU	Video Nvidia GTX 650 – 1GB
Motherboard	Gigabyte 970A-D3
Power Supply	ATX 650W Real
Operational System	Linux Ubuntu 14.04.4 LTS

Proteins in the last Critical Assessment of protein Structure Prediction (CASP), hosted in 2016,6 were studied for the construction of a strong background. CASP is one of most important events regarding PSP and the data used by the groups involved are a good point of investigation. Below are two histograms with the protein sizes considered Figure 6 shows the target proteins used as goals and disputed by the groups with the best predictions. Figure 7 displays the Domain Definition proteins and their sizes exhibit a similar frequency.

Figure 6

Histogram of the proteins size processed at the last CASP, in 2016. Those are the proteins targets.

Figure 7

Histogram of the proteins size processed at the last CASP, in 2016. Those are the domain definitions.

A diverse set of parameters was defined for the experiments, also varying the protein chain sizes. 2PG Mono [41] is the predictor chosen that uses a genetic algorithm to define the protein objective form. Besides protein size, the two other parameters are population size and number of generations for the genetic algorithm. They were chosen without a pattern towards making their prediction harder. Table 4 shows the respective values.

Table 4

Proteins and parameters for the experiments. Database: PDB.

Protein	Chain Size	Population	Generations
PDB:2N0L	12	From 20 to 30	From 10 to 150
PDB:1BCV	20	From 20 to 30	From 10 to 150
PDB:1AI0	21	From 20 to 500	From 20 to 1000
PDB:2ETI	28	From 20 to 200	From 20 to 400
PDB:1C94	38	From 20 to 200	From 20 to 1000
PDB:5K2L	49	From 20 to 200	From 40 to 800
PDB:5HVZ	50	From 20 to 100	From 10 to 400
PDB:1B1G	75	From 20 to 50	From 40 to 400
PDB:1EOD	100	From 100 to 300	From 200 to 1200
PDB:1CM7	100	From 30 to 100	From 30 to 100
PDB:1OZ9	150	From 20 to 50	From 30 to 200
PDB:4IEU	200	From 50	From 2 to 30
PDB:1AGY	200	From 20 to 50	From 30 to 300
PDB:2EEK	220	From 20 to 30	From 20 to 200
PDB:2R3A	300	From 20	From 60 to 200
PDB:2BX6	350	From 20 to 50	From 40 to 150

In a first approach, the experiment was conducted in a homogeneous set of machines for a better understanding of the behavior of the workload and the differences imposed by the change in the parameters. The results are discussed in the following charts. Figure 8 shows the overall processing time for all proteins. The curve displays a growing tendency, according to the parameters input. The higher parameters in the last columns show the impact of the larger chains on the system. Proteins of this size are common in many PSP experiments, therefore, systems should be prepared for them.

Figure 8

Processing time results for all scenarios.

For a better perspective on processing time growth, protein PDB:1EOD is not included in Figure 9, because the load it imposes on the system distorts the scale. The chart shows the differences in the processing time of the proteins. Some may appear more than once in the X axis, as the parameters of population and generation change.

Figure 9

Processing time results after the exclusion of protein PDB:1EOD.

Figure 10 displays the largest processing time for each protein. The workload overhead imposed on the infrastructure ranges from minutes to several hours. Even smaller initial chain sizes may be in the process for long times, i.e., up to days.

Figure 10

Larger processing times recorded.

Three proteins, namely PDB:2N0L (12 amino-acids), PDB:1B1G (75 amino-acids) and PDB:1AGY (200 amino-acids), were chosen for a comparison of samples according to their sizes. The aim was to present not only the differences between the input parameters, but also the way machines of lower capacity can deliver good results if the workload is suitable. The differences are shown Figure 11.

Figure 11

Sample of three PSP calculations. The results may overlap the higher classes, which hampers the prediction of the system's service capacity.

The information enabled some data evaluations for the definition of a general model that correctly represents the relations among experimental parameters, machine capacities and response time. As discussed in sections 4 and 5, lower-capacity machines fulfill users' requirements. Once Galaxy is adaptable to a variety of computational solutions the working machines were divided into three different classes, namely desktop computer, cluster machine and private cloud inside LASDPC. The different configurations for the machines are shown in Table 2, Section 4.2. The same experimental parameters of Table 4 were used, but in this case, different machine classes were chosen according to the input. The experiments were conducted in Koala, as reported in Section 4, and created a database of historic executions used for the regression models. The algorithms chosen for the regression are discussed in Section 4.2.

First, the data were loaded and a target variable was defined. Once, in this case, “Processing time” was the target, the best time for the execution of a protein in a selected machine could be estimated. Some information, such as protein names and resulting file sizes were discarded from the database because the regression tests did not use it. The data were then divided into two sets, namely train and test [44].

A few trials were necessary for achieving the modeling that suited the type of data collected. Three linear regression techniques were experimented and unsatisfactory results were yielded. Each model was tested with four types of data transformation, namely raw data, normalized data, log and normalized log. A few transformations in the raw variables, as conversion of strings to numerical values and scaling of numerical variables to mean = 0 and standard deviation = 1, were required [44].

The first experiment was the basic Linear Regression, however, the error rate was unsatisfactory [45]. Least Absolute Shrinkage and Selection Operator (Lasso) Regression were then tested, with L1 penalty, however, again, the results did not fulfill the requirements for a good model [46]. Ridge Regression with L2 Penalty [47] was the third regression to be tested and also yielded inadequate results. The general results for the models are shown in Table 5.

Table 5

General results for the linear models over the collected data. Labels: LR (Linear Regression), LL (Lasso Linear) and RR (Ridge Regression). R Squared: Ideal is closest to 1. RMSE (Root Mean Squared Error): The less the better.

Model	Score Function	Raw	Normalized	Log	Normalized Log
LR	R Squared	0.48	0.48	0.53	0.53
LR	RMSE	932.72	932.72	881.05	881.05
LL	R Squared	0.48	0.49	0.53	0.53
LL	RMSE	930.74	920.41	880.91	880.32
RR	R Squared	0.48	0.49	0.54	0.54
RR	RMSE	931.53	924.74	875.63	875.44

A non-linear model was applied to the data and results analyzed. Support Vector Regression (SVR) is a variation of Support Vector Machines (SVM) used for classification, regression models and construction of strong models. SVMs were initially developed for the binary classification of numeric data. They focus on the optimization of the hyper-plane that maximizes the margin between the two classes, which is proven to generalize well to unseen data as it is made in the framework of statistical learning theory [42]. SVR was implemented with the train set and its performance was assessed by predicting never seen data using the same data set. The following score functions were defined for the modeling of SVR:

(R-squared): as close to 1 the output, the better.

RMSE (Root Mean Squared Error): the lesser, the better.

The overall results were superior than those of linear regressions. As shown in Table 6, SVR achieved R squared results close to the ideal ones and different RMSE numbers. The best results were provided by the normalized data, with a small deviation.

Table 6

Results of the running of SVR model on the collected data. Best results highlighted.

Model	Score	Raw	Normalized	Log	Normalized Log
SVR	R Squared	0.98	0.997020	0.94	0.95
SVR	RMSE	174.64	70.505679	312.44	293.05

Figure 12 displays the predicted values obtained by the SVR plotted versus true values. The points follow a diagonal line, with few variability, which has proven that the predictions are very close to the expected values.

Figure 12

Results from SVR learning algorithm. The accuracy is high, with few deviations.

The model generated is now feasible to try the SVR with information not available in the data-set and have a prediction of the processing time for each environment. By adding the parameters as any set of the experiment and informing the three types of machine available, the model estimates the processing time for each one and indicates the best option. The three environments were tested with the same parameters below:

Protein chain size = 72 amino-acids,

Population size = 80 individuals,

Generations = 200.

Figure 13 shows the results. The desktop environment displayed the worst results, while cluster and cloud machines exhibited similar response times. The cluster machine provided a slightly better result than the cloud, which has proven local environments can properly attend users in some cases. Even in a tied result, the use of the cloud must be avoided, due to bandwidth and costs involved.

Figure 13

Results predicted by the SVR model on never seen data. The best result in this sample was achieved by the cluster machine.

The model predicts the ideal environment for each PSP experiment defined in the scenario and Decision Maker can correctly determine the workload that fits a machine configuration. Considering the complete data-set, the best type of machine according to the workload is observed in the Figure 14. The almost equal job distribution and the less powerful machine (desktop) chosen in 41% of the experiments show the necessity of an evaluation of each case for the avoidance of waste of resources and computing capacity.

Figure 14

Division of jobs according to proteins workload for the corresponding machine capacity.

According to the results, best fits can be defined for experiments with no overload of a machine's capacity and the purchase of over qualified (and expensive) hardware is not necessary for the running of experiments. Moreover, investments in local hardware, either independent hosts or clusters, are advantageous for the development of research. Cloud computing has many conveniences, but does not work best to every case. We consider local management, setup freedom and avoidance of bandwidth limitations are a strong defense of the installation of local systems and enable remote access, when necessary. Paying for cloud solutions on a long term may be a wrong choice, as that requires high investments.

Related work

Performance and user's experience have been addressed in scientific platforms, both on structural level and software implementations. Techniques that improve quality and reduce limitations involve diverse experimentation aspects.

Moreno et al. [48] developed a library for the processing of long protein sequences. The experiments revealed the bioinformatics bottleneck had changed from data acquisition to data interpretation.

Thaman and Singh [49] reviewed the services offered in distributed architectures and the task scheduling was identified as the most influential factor for the extraction of computational resources performance. Shagwan and Kumar [50] conducted an extensive review of scheduling algorithms for clouds. Both studies support our decision of applying machine learning on the Decision Maker.

Bianchi et al. [51] introduced a workflow management system to address Next Generation Sequencing (NGS). However, the authors did not consider cluster and cloud infrastructures for running the experiments and the solution is closed and not adaptable to other applications. Akos et al. [3] proposed a generic Science Gateway where applications can be uploaded as black box components. Although it can be used as a basis for new portals, it is not an optimized solution and might lead to poor performance. Kacsuk [52] described a portal for the integration of tools that use grid systems for the execution of tasks. Although it is no longer available, it could be used for evaluation purposes.

Stitz et al. [53] proposed a solution to deal with the lack of patterns on virtualized nodes. It is a visualization system that supports the arrangement of the computational resources offered. However, the study did not address the methods appropriately used and failed to consider a variety of machines. Liu et al. [54] integrated Galaxy with Globus Transfer for a reliable data moving. The resources are provisioned on-demand, however, they cover exclusively paid cloud machines and do not fit any user. The issues addressed by such studies are common in distributed systems. Different technologies and strategies are tested to solve scalability, scheduling, monitoring, and heterogeneous platforms problems.

Sandes et al. [9] reviewed the architectures from FPGAs to GPU applied to bioinformatics and compared solutions that best fitted each architecture. The study showed the necessity of highly parallel solutions and platform variations. Mrozek et al. [55] introduced Cloud4PSP, a solution that adopts a scalability model to run ab initio proteins in clouds. It is based on the pay-as-you-go model and relies on horizontal and vertical scalability for enhancing the performance of the predictions. However, it is a closed solution strictly tied to a company.

Karoczkai et al. [56] presented a meta-broker for science gateways for scheduling jobs to heterogeneous machines. It creates a layer on the gUSE gateway to set job priorities and distributes them according to the distance from of the service providers. Although it can be a solution to load balancing, it does not fit the workload in available resources, which might impair its performance. Sandes et al. [57] filled this gap proposing an implementation that considers the user's expertise to provide wavefront balancing for a multinode arrangement. The solution maintains fair job shares, depending on the amount of data transfer and communication, and could be useful in applications that require less computational power.

Sandes et al. [19] proposed an extension of CUDAlign [31]. It consists of a multi-platform implementation for sequence alignment (MASA) that considers processing on GPU, FPGA, and CellBe architectures. It applies an optimization for reducing the number of cells calculated without losing precision and uses heterogeneous environments; however, it addresses smaller workloads.

Macedo et al. [58] proposed an allocation policy based on master/slave strategy for heterogeneous multi-core clusters. Their study showed a proper placement of master nodes can reduce processing time. The experiments were limited by the cluster capacities and did not consider scaling down of the machines for smaller requirements.

Science gateways are multi-tenant systems that can benefit from specific provision policies. Peng et al. [59] described issues related to the maintenance of multi-tenant systems and introduced a knowledge-based resource allocation manager. The results showed reasonable execution time, however, the manager module produces additional costs.

Ying and Lei [60] designed a dynamic scheduler for multi-tenant systems that uses a mechanism based on preconditions for improving scheduling and reducing execution time. However, the approach must be tested in wider systems under harder conditions.

Garza et al. [18] introduced a set of tools for workflow scheduling. The idea is to distribute small workflow tasks to heterogeneous machines. It has opened up a new perspective for the design and execution of workflows, however, it does not consider the workload influence and its relation to the computing capacity.

This section has addressed the gaps on Scientific Gateways and integration research. Table 7 summarizes the aspects and differences among the related works and our novel solution for comparison purposes. Decision Maker is a solution based on execution history for properly modeling workload, according to users' claims and the literature. It suggests a scalable and flexible environment for workflow execution considering reproducibility and budget. To the best of our knowledge, Decision Maker is a good contribution to those systems and could be a powerful tool for improving the execution and provision of computational resources.

Table 7

Evaluation of related works.

Work	Platform	Scalable	Reproducibly	Flexible	Economy
Akos et al. [3]	Grid	Yes	No	No	Yes
Liu et al. [54]	Cloud	Yes	Yes	No	No
Peng et al. [59]	Cloud	No	No	Yes	Yes
Ying and Lei [60]	Cloud	Yes	No	No	Yes
Bianchi et al. [51]	Cloud	No	Yes	No	No
Karoczkai et al. [56]	Multi	Yes	Yes	No	No
Sandes et al. [19]	Multi	No	No	Yes	Yes
Sandes et al. [9]	Multi	No	No	Yes	No
Garza et al. [18]	Multi	Yes	Yes	Yes	No

Conclusions

Scientific Gateways have become a handy tool to support researchers regarding experiment execution, data storage, and dissemination of results. However, in many cases systems show limitations, such as low processing capacity, storage space and response time. Such problems configure capacity bottlenecks, which compromise the efficiency of the systems.

Based on Galaxy experiments, we evaluated the retrieved information and extracted a general model from the data by using different regression techniques. We have proposed a decision module for improving the flexibility of execution environments and offering a smooth transition between multiple environments. The solution can also adapt to independent user's investment capabilities and avoid waste of resources. The module and its unique machine learning application is the differential component of infrastructure that supports current environments.

The SVR model showed high accuracy in predicting the proper machine to execute the workflow based on user's parameters definition and a good estimation of the processing time. A varied environment can be the best solution, once it includes cloud computing and offers other types of machines.

As future work, we intend to model an architecture on WorkflowSim for a complete view of the science gateway capacity, considering the whole network, connections and modules involved. Decision Maker is on the core of the architecture, towards aiding the definition of the best machine capacity to scientific experiments.

Declarations

Author contribution statement

Edvard M. de Oliveira: Conceived and designed the experiments; Performed the experiments; Wrote the paper.

Júlio C. Estrella: Analyzed and interpreted the data; Wrote the paper.

Alexandre C.B. Delbem: Conceived and designed the experiments; Wrote the paper.

Luiz H. Nunes, Henrique Y. Shishido, Stephan Reiff-Marganiec: Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This work was supported by the Brazilian National Council for Scientific and Technological Development (CNPQ) (165009/2015-2).

Competing interest statement

The authors declare no conflict of interest.

Additional information

Data associated with this study has been deposited at http://bit.ly/protein-algos.