If you draw a diagonal line with square pixels, their hard edges create a jagged ‘staircase’ effect. This ugliness (among other artifacts) is called aliasing. If resolutions were much higher, it wouldn’t be a problem, but until display technology advances, we have to compensate with anti-aliasing.
There are many techniques for anti-aliasing, but supersampling (SSAA) is useful to explain the process. It works by rendering frames at a higher resolution than the display resolution, then squeezing them back down to size. On the previous page, you can see the anti-aliasing effect of downsampling Shadow of Mordor from 5120x2880 to 1440p.
Consider a pixel on a tile roof. It’s orange, and next to it is a pixel representing a cloudy sky, which is light and blueish. Next to each other, they create a hard, jagged transition from roof to sky. But if you render the scene at four times the resolution, that one orange roof pixel becomes four pixels. Some of those pixels are sky-colored and some are roof-colored. If we take the average of all four values, we get something in between. Do that to the whole scene and the transitions become softer.
That's the gist, at least, and while it looks very good, supersampling is extremely computationally expensive. You’re rendering each frame at a resolution two or more times higher than the one you’re playing at—even with our multiple high-end graphics cards, trying to run supersampling with a display resolution of 2560x1440 isn't practical. That’s why there are so many more efficient alternatives:
Multisampling (MSAA): More efficient than supersampling, but still demanding. This is typically the standard, baseline option in older games, and it's explained very simply in the video below.
Coverage Sampling (CSAA): Nvidia’s more efficient version of MSAA. You won't see this much anymore.
Custom-filter (CFAA): AMD’s more efficient version of MSAA. You also won't see this much anymore.
Fast Approximate (FXAA): Rather than analyzing the 3D models (i.e. MSAA, which looks at pixels on the edges of polygons), FXAA is a post-processing filter, meaning it applies to the whole scene after it has been rendered, and it's very efficient. It also catches edges inside textures which MSAA misses. This is the default in many modern games because it has very little overhead, though it tends to miss a lot of jaggies.
Morphological (MLAA): Available with AMD cards, MLAA also skips the rendering stage and processes the frame, seeking out aliasing and smoothing it. As Nicholas Vining explains: "Morphological anti-aliasing looks at the morphology (read: the patterns) of the jaggies on the edges; for each set of jaggies, it computes a way of removing the aliasing which is pleasing to the eye. It does this by breaking down edges and jaggies into little sets of morphological operators, like Tetris blocks, and then uses a special type of blending for each Tetris block." MLAA can be enabled in the Catalyst control panel.
Enhanced Subpixel Morphological (SMAA): Another post-processing method, described as combining MLAA with MSAA and SSAA strategies. You can apply it with SweetFX, and many modern games natively support this.
Temporal (TAA or TXAA): TXAA was initially supported on Nvidia's Kepler and later GPUs, but more general forms of temporal anti-aliasing are now available and are typically just labeled TAA. TAA compares the previous frame with the current frame to look for edges and help remove jaggies. This is done through a variety of filters and can help reduce the 'crawling' motion on edges, which looks a bit like marching ants. It cannot, however, remove actual ants from inside your display. You should probably just throw that display out.
As Vining again explains: "The notion here is that we expect frames to look a lot like each other from frame to frame; the user doesn't move that much. Therefore, where things haven't moved that much, we can get extra data from the previous frame and use this to augment the information we have available to anti-alias with."
Multi-Frame (MFAA): Introduced with Nvidia's Maxwell GPUs. Whereas MSAA samples in set patterns, MFAA allows for programmable sample patterns. Nvidia does a good job of explaining MSAA and MFAA simply in the video below.
Deep Learning Super-Sampling (DLSS): Nvidia's latest, available on certain games using new GeForce RTX cards. According to Nvidia, "DLSS leverages a deep neural network to extract multidimensional features of the rendered scene and intelligently combine details from multiple frames to construct a high-quality final image. DLSS uses fewer input samples than traditional techniques such as TAA, while avoiding the algorithmic difficulties such techniques face with transparency and other complex scene elements."
In other words, it's better and more efficient at doing the things Temporal AA does, or at least it's supposed to look better once it has been properly trained for a specific game. When it's not properly trained, it can cause a lot of blurriness.
What do the numbers mean?
Anti-aliasing settings almost always include a series of values: 2x, 4x, 8x, and so on. The numbers refer to the number of color samples being taken, and in general, the higher the number, the more accurate (and computationally expensive) the anti-aliasing will be.
Then there's the special case of the 'Q.' CSAA attempts to achieve a quality better than or equal to MSAA with fewer color samples, so 8xCSAA actually only takes four color samples. The other four are coverage samples, explained here. 8QxCSAA, however, bumps the number of color samples back up to eight for increased accuracy. You'll rarely encounter CSAA these days, so let's call that a fun fact.
I used Batman: Arkham City's benchmarking tool to test a few older anti-aliasing methods: MSAA, FXAA, and TXAA. The results show, as expected, that FXAA is the least resource intensive, while MSAA and TXAA cause a significant drop in average framerate over no anti-aliasing.
Batman: Arkham City AA benchmarks (2x Nvidia GTX Titan SLI)
Some more modern games don't give you much of a choice. Battlefield 5, for example, offers me two TAA quality options. Assassin's Creed Odyssey offers low, medium, high, and adaptive options.
Which AA should I use?
It depends on your GPU, your preference, and what kind of performance you're after. If framerate is an issue, however, the choice is usually obvious: FXAA is very efficient. If you've got an RTX card, and the game you're playing supports it, give DLSS a try—you paid for it, and it's top of the line. In older games, you'll probably have to do a bit of testing to get the combination of look and performance you want. If you have the hardware to do it, you can also try supersampling instead of using the built-in options, which usually works. Overriding settings in other ways, however, isn't a sure thing.
Overriding anti-aliasing settings
Theoretically, in-game graphics options shouldn’t matter. You can just open up the Nvidia or AMD control panel and override their settings. Unfortunately, that’s not really the case. While you can set overrides for any game, I've had little success getting them to work.
"Very often when overrides don't work it's due to deferred rendering," says Vining, "which just breaks a lot of common anti-aliasing techniques." Alex Austin also notes that some of his techniques don't work with override settings. So, it's just a matter of testing. Turn off all AA in the in-game options, set the override in your control panel, and hop back in: it should be apparent whether or not it took effect.
I've found that AMD's MLAA works the most when enabled in the control panel. It's important to note, however, that it's a post-processing filter and applies to everything in the scene. That means it can take care of hard edges within textures, which can be good, but comes with the side-effect that it may also go after desirable edges, such as in text. Notice how it slightly smooths the text in the BioShock Infinite menu, and even goes so far as to smooth my FRAPS FPS overlay.
Supersampling using Nvidia's DSR or AMD's Virtual Super Resolution, however, is more reliable. In the AMD settings, you simply have to turn Virtual Super Resolution on in the Display tab. If you have an Nvidia card, head to the 'Manage 3D settings' section of the Nvidia Control Panel, where you can select DSR factors up to 4x. Do one of those things, and in your games' options you should be able to select resolutions higher than your display resolution. Do so, and the game will run at the selected resolution and be downsampled to your display resolution—very taxing, but it looks nice. This can cause UI problems in some games, or just not work at all.