Thomas Trolle1, Imir G Metushi1, Jason A Greenbaum1, Yohan Kim1, John Sidney1, Ole Lund1, Alessandro Sette1, Bjoern Peters1, Morten Nielsen2. 1. Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and Instituto de Investigaciones Biotecnológicas, Universidad Nacional de San Martín, San Martín, Buenos Aires, Argentina. 2. Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and Instituto de Investigaciones Biotecnológicas, Universidad Nacional de San Martín, San Martín, Buenos Aires, Argentina Center for Biological Sequence Analysis, Department of Systems Biology, The Technical University of Denmark, Lyngby, Denmark, Division of Vaccine Discovery, La Jolla Institute for Allergy and Immunology, La Jolla, CA 92037, USA and Instituto de Investigaciones Biotecnológicas, Universidad Nacional de San Martín, San Martín, Buenos Aires, Argentina.
Abstract
MOTIVATION: Numerous in silico methods predicting peptide binding to major histocompatibility complex (MHC) class I molecules have been developed over the last decades. However, the multitude of available prediction tools makes it non-trivial for the end-user to select which tool to use for a given task. To provide a solid basis on which to compare different prediction tools, we here describe a framework for the automated benchmarking of peptide-MHC class I binding prediction tools. The framework runs weekly benchmarks on data that are newly entered into the Immune Epitope Database (IEDB), giving the public access to frequent, up-to-date performance evaluations of all participating tools. To overcome potential selection bias in the data included in the IEDB, a strategy was implemented that suggests a set of peptides for which different prediction methods give divergent predictions as to their binding capability. Upon experimental binding validation, these peptides entered the benchmark study. RESULTS: The benchmark has run for 15 weeks and includes evaluation of 44 datasets covering 17 MHC alleles and more than 4000 peptide-MHC binding measurements. Inspection of the results allows the end-user to make educated selections between participating tools. Of the four participating servers, NetMHCpan performed the best, followed by ANN, SMM and finally ARB. AVAILABILITY AND IMPLEMENTATION: Up-to-date performance evaluations of each server can be found online at http://tools.iedb.org/auto_bench/mhci/weekly. All prediction tool developers are invited to participate in the benchmark. Sign-up instructions are available at http://tools.iedb.org/auto_bench/mhci/join. CONTACT: mniel@cbs.dtu.dk or bpeters@liai.org SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.
MOTIVATION: Numerous in silico methods predicting peptide binding to major histocompatibility complex (MHC) class I molecules have been developed over the last decades. However, the multitude of available prediction tools makes it non-trivial for the end-user to select which tool to use for a given task. To provide a solid basis on which to compare different prediction tools, we here describe a framework for the automated benchmarking of peptide-MHC class I binding prediction tools. The framework runs weekly benchmarks on data that are newly entered into the Immune Epitope Database (IEDB), giving the public access to frequent, up-to-date performance evaluations of all participating tools. To overcome potential selection bias in the data included in the IEDB, a strategy was implemented that suggests a set of peptides for which different prediction methods give divergent predictions as to their binding capability. Upon experimental binding validation, these peptides entered the benchmark study. RESULTS: The benchmark has run for 15 weeks and includes evaluation of 44 datasets covering 17 MHC alleles and more than 4000 peptide-MHC binding measurements. Inspection of the results allows the end-user to make educated selections between participating tools. Of the four participating servers, NetMHCpan performed the best, followed by ANN, SMM and finally ARB. AVAILABILITY AND IMPLEMENTATION: Up-to-date performance evaluations of each server can be found online at http://tools.iedb.org/auto_bench/mhci/weekly. All prediction tool developers are invited to participate in the benchmark. Sign-up instructions are available at http://tools.iedb.org/auto_bench/mhci/join. CONTACT: mniel@cbs.dtu.dk or bpeters@liai.org SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.
Authors: Xiao-Fei Wang; William M Cohen; Florence A Castelli; Christine Almunia; Bernard Lethé; Sandra Pouvelle-Moratille; Gaetan Munier; Dominique Charron; André Ménez; Hassan M Zarour; Pierre van der Bruggen; Marc Busson; Bernard Maillère Journal: Cancer Immunol Immunother Date: 2006-09-20 Impact factor: 6.968
Authors: Thomas Trolle; Curtis P McMurtrey; John Sidney; Wilfried Bardet; Sean C Osborn; Thomas Kaever; Alessandro Sette; William H Hildebrand; Morten Nielsen; Bjoern Peters Journal: J Immunol Date: 2016-01-18 Impact factor: 5.422
Authors: Abraham Sachs; Eugene Moore; Zeynep Kosaloglu-Yalcin; Bjoern Peters; John Sidney; Steven A Rosenberg; Paul F Robbins; Alessandro Sette Journal: J Immunol Date: 2020-06-22 Impact factor: 5.422
Authors: Hadi Karimzadeh; Muthamia M Kiraithe; Anna D Kosinska; Manuel Glaser; Melanie Fiedler; Valerie Oberhardt; Elahe Salimi Alizei; Maike Hofmann; Juk Yee Mok; Melanie Nguyen; Wim J E van Esch; Bettina Budeus; Jan Grabowski; Maria Homs; Antonella Olivero; Hossein Keyvani; Francisco Rodríguez-Frías; David Tabernero; Maria Buti; Andreas Heinold; Seyed Moayed Alavian; Tanja Bauer; Julian Schulze Zur Wiesch; Bijan Raziorrouh; Daniel Hoffmann; Antonina Smedile; Mario Rizzetto; Heiner Wedemeyer; Jörg Timm; Iris Antes; Christoph Neumann-Haefelin; Ulrike Protzer; Michael Roggendorf Journal: J Virol Date: 2018-06-13 Impact factor: 5.103