You are offered a selection of maps to search (you can only choose one at a time) and a selection of molecules that act as a mask to this map. Finally you must choose which ligand types you are going to search for in this map 1. Only molecules with less than 400 atoms are suggested as potential ligands.
If you do not have any molecules with less that 400 atoms loaded in Coot, you will get the message:
"Error: you must have at least one ligand to search for!"
New ligands are placed where the map density is and protein (mask) atoms are not). The masked map is searched for clusters using a default cut-off of 1.0\sigma. In weak density this cut-off may be too high and in such a case the cut-off value can be changed using something such as:
(set-ligand-cluster-sigma-level 0.8)
However, if the map to be searched for ligands is a difference map, a cluster level of 2.0 or 3.0 would probably be more appropriate (less likely to generate spurious sites).
Each ligand is fitted with rigid body refinement to each potential ligand site in the map and the best one for each site selected and written out as a pdb file. The clusters are sorted by size, the biggest one first (with an index of 0). The output placed ligands files have a prefix “best-overall” and are tagged by the cluster index and residue type of the best fit ligand in that site.
By default, the top 10 sites are tested for ligands - to increase this use:
(set-ligand-n-top-ligands 20)
If the “Flexible?” checkbutton is activated, coot will generate a number of variable conformations (default 100) by rotating around the rotatable bonds (torsions). Each of these conformations will be fitted to each of the potential ligand sites in the map and the best one will be selected (again, if it passes the fitting criteria above).
Before you search for flexible ligands you must have read the mmCIF dictionary for that particular ligand residue type (File -> Import CIF dictionary). Use:
(set-ligand-flexible-ligand-n-samples n-samples)
where n-samples is the number of samples of flexibility made for each ligand. Generally speaking, The more the number of rotatable bonds, the bigger this number should be.
By default the options to change these values are not in the GUI. To enable these GUI options, use the scripting function:
(ligand-expert)
After successful ligand searching, one may well want to add that displayed ligand to the current model (the coordinates set that provided the map mask). To do so, use Merge Molecules (Section Merge Molecules).