Measuring the performance of ligand-based methods

This is another episode in the "how good are these methods really?" series. The aim is to understand how well the ligand-based 3D methods for virtual screening perform under standard benchmarking sets. In the original papers introducing the methods frequently is benchmarking done on a small test sets, prone to all sorts of small population problems. 

Here I take 3 popular methods: the ROCS-like implementation of Gaussian shape overlap[1] that I recently ported to rdkit as well as USR[2] and USR-CAT[3], two very fast shape methods. All 3 are tested on the DUDE[4] dataset, a standard benchmarking set. Conformer generation was done as outlined by JP in his paper[5].



Pair of methods compared Statistic used Value
rdkit-shape VS usr wilcoxon T=2.24, p=0.02
rdkit-shape VS usrcat wilcoxon T=-0.47, p=0.63
usr VS usrcat wilcoxon T=-2.70, p=0.006

One thing that struck me was the spread of performance. I got a little worried that the conformer generation method was messed up, so I tested that part and it correctly reproduced the crystal poses to within 2-3 Angstrom easily (below). So that part is okay.

It still could be that there is a numerical problem with rdkit-shapeit – and that it can't find the correct maximum overlap but I'm not sure how to test for that. 

But let's take it all at a face value. These methods are not great... USR is worst of all, kind of making a case that it doesn't matter if the method is fast, it matters if it gives any sort of sensible results. rdkit-shapeit isn't much better. USRCAT performs best of all, along the lines of "if you include some chemistry features, it works much better". It seems that although shape complimentary is a necessary condition for molecular similarity, it's certainly not a sufficient condition...

Going back to the general statement (that these methods don't look awesome), it's clear that even the best one offers a median AUC (0.6) that's barely above random (0.5). The interquartile range for all 3 methods is huge, which means that depending on your target protein they may work well or terrible. It's interesting to think that a method with a narrow interquartile range around AUC of 0.6 would be more desirable that a method with median AUC of say 0.7 but with huge variation from target to target. 

A blank space is left for rdkit-alignit, a pharmacophore tool. rdkit-alignit is to rdkit-shape what usrcat is to usr – it incorporates chemistry features into the search. The blank space is left because I haven't ported alignit into rdkit yet and so I can't make the comparison.

References 
[1] http://onlinelibrary.wiley.com/doi/10.1002/%28SICI%291096-987X%2819961115%2917:14%3C1653::AID-JCC7%3E3.0.CO;2-K/abstract
[2] http://www.ncbi.nlm.nih.gov/pubmed/17342716
[3] http://www.jcheminf.com/content/4/1/27
[4] http://pubs.acs.org/doi/abs/10.1021/jm300687e
[5] www.ncbi.nlm.nih.gov/pubmed/22482737

Comments

Post a Comment

Popular posts from this blog

React.js – edit and delete comments in CommentBox

It appears customary for high-intelligence, high-skills groups of engineers to "roll their own" whenever they have a chance to. Counter-intuitively to most managerial/MBA types out there, re-inventing the wheel can give you... a better wheel. Success is not guaranteed though – but these people may leave sooner rather than later if you don't let them... Having looked at number of JS frameworks (ember, backbone, angular) there was no clear winner. They were all a mess, at least to me. So I approached React.js with no hopes – how much better could it possible be? It turns out, a lot! This post will be an extension of the official React.js tutorial . The tutorial stops at the stage where you can add comments to a dynamic lists, that will grow as you add comments. This falls short of the functionality displayed on facebook: once you add a comment you can edit or delete it, so that's what're going to do. Full disclosure, I only started using React this weekendo so m...
Read more

Example slurm cluster on your laptop (multiple VMs via vagrant)

To anybody who's considering a switch to modern queuing systems, like slurm, this will be an useful guide. Rather than doing a roll out on production nodes, we'll try things out in vagrant. This text assumes you have vagrant installed on your local machine (it's very easy). The ultimate aim will be to setup 2 servers (worker nodes) and one master/controller, and run a simple job on them. There are multiple great guides out there: most of them are out of date, and won't work without some tweaking on Ubuntu 16.04 Xenial. The entirety of the code is hosted at https://github.com/jandom/gromacs-slurm-openmpi-vagrant The README.md file will show you how to set it up, with checks on every step. References https://mussolblog.wordpress.com/2013/07/17/setting-up-a-testing-slurm-cluster/ https://github.com/gabrieleiannetti/slurm_cluster_wiki/wiki/Installing-a-Slurm-Cluster http://philipwfowler.me/2016/04/14/how-to-setup-a-gramble/ https://github.com/dakl/slurm-clus...
Read more

Wrapping openbabel in python - using cython

The previous post showed a contrived example of how one could access the openbabel functionality in python using the boost libraries. There is alternative to boost to wrap around c/c++ code, it's called cython. Here is an example openbabel wrapper, exposing some very simple functionality. Just before diving in, you should know that there already exists a wrapper around openbabel in python (it is called pybel). What I'm showing here doesn't even come close to pybel in terms of usefulness, the idea here is to show a demo of how easy wrapping things in python is. #! /usr/bin/python from ez_setup import use_setuptools use_setuptools() from setuptools import setup, Extension, find_packages from Cython.Distutils import build_ext import sys, os import glob import subprocess as sub def get_include(name="openbabel"): p = sub.Popen('locate %s' % name ,stdout=sub.PIPE,stderr=sub.PIPE, shell=True) output, errors = p.communicate() ...
Read more