Dynamic benchmarking in bioinformatics

Abstract

In bioinformatics there often exist a wide selection of different algorithms to solve a given problem. Therefore, benchmarks are often used by researchers to either find the most suitable algorithms for their specific need, or compare their newly developed algorithm to existing ones. Though benchmarks can be useful for finding the right algorithm for the problem, good and relevant benchmarks are often lacking in many areas. This is due to the fact that benchmarks tend to be quite difficult and time consuming to develop, and it’s hard to keep up to date with new developments. We here present an online benchmarking system, integrated in a Galaxy-based platform. This is a system aimed at biologists with little or no programming experience as well as bioinformaticians who don’t want to build benchmark functionality from scratch. The purpose is to provide a tool that enables users to easily create and share benchmarks and evaluate test sets that reflects the users specific needs. The tool streamlines each step of the benchmarking process, and supports benchmarking for a selection of different problem areas such as gene prediction, motif discovery, splice site prediction and nucleosome prediction. The system was evaluated with six different test sets, assessing the systems capability of handling benchmarks across different problem areas, sizes and types. Despite some current limitations, the system shows promise as a potential platform for benchmarking in bioinformatics. The system is available for use at http://hyperbrowser.uio.no/bench/, and consists of three tools; Benchmark Creation Tool, Benchmark Retrieval Tool and Benchmark Evaluation Tool.