Introduction

The strengths and weaknesses of different methods for gene network inference remain poorly understood. On the one hand, this can be explained by the difficulty of constructing adequate benchmarks, and on the other hand, by the lack of tools for a differentiated analysis of network predictions on such benchmarks. Below, we summarize our work to address these two issues:

• Generating realistic in silico benchmarks
• Revealing strengths and weaknesses of inference methods

Generating realistic in silico benchmarks

Evaluating the performance of methods for inference of gene regulatory networks is difficult, because network predictions can in general not be systematically tested in vivo with the current technology. Consequently, in silico benchmarks based on simulated networks and expression data are essential to assess the performance of inference methods (Figure 1).

We are developing tools for the generation of biologically plausible benchmarks, which enable realistic in silico performance assessment:

• We generate realistic network structures for the benchmarks by extracting modules from known gene networks of model organisms, instead of using random graph models as commonly done. [2]

• We endow these networks with dynamics using a kinetic model that is more detailed than those of other in silico benchmarks. It is based on a thermodynamical model of transcriptional regulation, and it includes both mRNA and protein dynamics. Furthermore, in addition to measurement noise, we also model internal noise in the dynamics of the gene networks. [1]

This framework allows inference methods to be tested in silico on networks with similar types of structural properties, regulatory dynamics, and noise as occur in biological gene networks.

Figure 1. (A) The "true" network structure of biological gene networks is in general unknown, which hinders systematic performance evaluation. (B) For in silico networks, the ground truth is known and predictions can be validated.

Revealing strengths and weaknesses of inference methods

We have developed a novel approach to analyze network predictions. In addition to assessing the overall accuracy, we evaluate the performance on different types of local connectivity patterns (motifs) of the networks. We have used this approach to assess the performance of 62 network-inference methods that have been applied independently by participating teams in two community-wide reverse engineering challenges (DREAM3 and DREAM4 in silico challenges).

Our results show that current inference methods are affected, to various degrees, by three types of systematic prediction errors: the fan-out error, the fan-in error, and the cascade error (Figure 2). In contrast to evaluation of overall accuracy, the network motif analysis reveals the strengths and weaknesses of inference methods, thereby suggesting possible directions for their improvement. [1]

Figure 2. (A) The true connectivity of the motifs. (B) As an example, we show how the motifs were predicted on average by one of the top-three methods of the DREAM3 in silico challenge. This reveals three types of systematic prediction errors (C). The darkness of the links indicates their median prediction confidence.