When exporting an 8 bit After Effects composition to a lossy format, you may end up with gradients that show visible banding, and the lower the bit rate, the worse the banding.
You may get better results by upping the bit depth on the composition to 16 bits, but you still will be depending on the downsampling algorithm within the AE Render Queue, or Adobe Media Encoder. And you’re also stuck with the codecs exposed through those pieces of software.
If you want to use x264 to have finer control over your encode, especially at very low bit rates, and stay compatible with specific target devices, such as the iPhone or iPad, you will most likely face the problems of feeding 16 bit image sequences to x264 via mplayer, or use some unholy avisynth script, as x264 will not accept ProRes or any other higher bit rate format natively.
One possible solution for this problem is using the QuickTime x264Encoder component. It will work within After Effects’ own Render Queue, QuickTime Pro, or Compressor and will accept any input QuickTime accepts. So simply export your AE 16 bit sequence directly to H.264 using the x264Encoder component, or export it first to ProRes 4444, and feed it to x264 via the component, with a lot of x264′s options exposed.
Update: Surprisingly, the component also works reasonably well within Qmaster distributed encoding. You may want to experiment with multithreading within the component’s settings vs. multithreading in Qmaster, or both. Provided you have enough storage and RAM and a fast network, this works true wonders when encoding hours of video at a time.