The built-in antialiasing (for what it's worth) can be enabled using:
(set-do-anti-aliasing 1)
The default is 0 (off).
This can also be activated using Edit Preferences -> Others -> Antialiasing -> Yes.
If you have an nVidia graphics card, external antialiasing can be actived setting the environment variable __GL_FSAA_MODE. For me a setting of 5 works nicely and gives a better image than using Coot's built-in antialiasing.
Also for nVidia graphics card users, there is the application nvidia-settings:
Antialiasing Setting -> Override Application Settings and slide the slider to the right. On restarting Coot, it should be in antialias mode 1.