Marek S Wiewiórka1, Antonio Messina1, Alicja Pacholewska2, Sergio Maffioletti1, Piotr Gawrysiak1, Michał J Okoniewski1. 1. Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland. 2. Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Abstract
UNLABELLED: Many time-consuming analyses of next -: generation sequencing data can be addressed with modern cloud computing. The Apache Hadoop-based solutions have become popular in genomics BECAUSE OF: their scalability in a cloud infrastructure. So far, most of these tools have been used for batch data processing rather than interactive data querying. The SparkSeq software has been created to take advantage of a new MapReduce framework, Apache Spark, for next-generation sequencing data. SparkSeq is a general-purpose, flexible and easily extendable library for genomic cloud computing. It can be used to build genomic analysis pipelines in Scala and run them in an interactive way. SparkSeq opens up the possibility of customized ad hoc secondary analyses and iterative machine learning algorithms. This article demonstrates its scalability and overall fast performance by running the analyses of sequencing datasets. Tests of SparkSeq also prove that the use of cache and HDFS block size can be tuned for the optimal performance on multiple worker nodes. AVAILABILITY AND IMPLEMENTATION: Available under open source Apache 2.0 license: https://bitbucket.org/mwiewiorka/sparkseq/.
UNLABELLED: Many time-consuming analyses of next -: generation sequencing data can be addressed with modern cloud computing. The Apache Hadoop-based solutions have become popular in genomics BECAUSE OF: their scalability in a cloud infrastructure. So far, most of these tools have been used for batch data processing rather than interactive data querying. The SparkSeq software has been created to take advantage of a new MapReduce framework, Apache Spark, for next-generation sequencing data. SparkSeq is a general-purpose, flexible and easily extendable library for genomic cloud computing. It can be used to build genomic analysis pipelines in Scala and run them in an interactive way. SparkSeq opens up the possibility of customized ad hoc secondary analyses and iterative machine learning algorithms. This article demonstrates its scalability and overall fast performance by running the analyses of sequencing datasets. Tests of SparkSeq also prove that the use of cache and HDFS block size can be tuned for the optimal performance on multiple worker nodes. AVAILABILITY AND IMPLEMENTATION: Available under open source Apache 2.0 license: https://bitbucket.org/mwiewiorka/sparkseq/.
Authors: Jeffrey B S Gaither; Grant E Lammi; James L Li; David M Gordon; Harkness C Kuck; Benjamin J Kelly; James R Fitch; Peter White Journal: Gigascience Date: 2021-04-05 Impact factor: 6.524
Authors: Aidan R O'Brien; Neil F W Saunders; Yi Guo; Fabian A Buske; Rodney J Scott; Denis C Bauer Journal: BMC Genomics Date: 2015-12-10 Impact factor: 3.969
Authors: Xinjie Zhu; Qiang Zhang; Eric Dun Ho; Ken Hung-On Yu; Chris Liu; Tim H Huang; Alfred Sze-Lok Cheng; Ben Kao; Eric Lo; Kevin Y Yip Journal: BMC Genomics Date: 2017-09-22 Impact factor: 3.969