At the same time, overclocking graphics hardware can be a little bit dicier than overclocking the CPU. Unlike your processor, your graphics card actually has two different domains for you to overclock: the GPU itself, and the video memory. The overwhelming majority of graphics cards have thermally controlled fans that take monitoring heat out of the equation, though you'll be trading noise for performance. I also personally don't recommend raising voltage; while consensus is easy enough to find for CPUs on what a "safe voltage" is for an individual model, GPUs are a little trickier and a lot more complex.
Finally, as with CPUs, you're going to be dealing with a lottery when you overclock your graphics card. Some models are notoriously overclocker friendly, having a tremendous amount of performance headroom to play with, while others are pretty much pushed as far as they'll go out of the box. In my personal experience, I've found NVIDIA's graphics hardware to generally have more headroom to overclock than AMD's, but I also have a pair of GTX 580s that wouldn't budge an inch, so your mileage may vary.
When overclocking modern kit, you'll need software. Thankfully all of this can be done in Windows. On AMD's graphics cards, you just need to enable Overdrive in the Catalyst Control Center. With NVIDIA's cards, you'll have to download and install software; my personal favorite is eVGA's Precision X, but if you can't get that, MSI's Afterburner will also work fine. Finally, I've found the absolute best stability test for graphics hardware is the most recent version of Futuremark's 3DMark.
Both AMD and NVIDIA have minor overclocking allowances built into their kit: AMD's is PowerTune, while NVIDIA has Boost (listed as Power Target in Precision X). For either one, you'll want to max out their allowances; this is just a way for the graphics card to control how much power it consumes. Since we're overclocking, we don't really care about power consumption as much.
For AMD cards and some NVIDIA cards, the GPU clock itself can be set in software, while on modern high end NVIDIA cards, you set an offset. In both cases, this serves to increase the core clock of the graphics card. With most cards you'll want to push the GPU clock as far as it'll go first, but there are some, like the GeForce GTX 770, or almost any card with a 256-bit memory bus or smaller and GDDR5, where you'll want to prioritize overclocking the video memory. Note that overclocking one can actually limit how far you can push the other. Do not overclock both at the same time; you want to control for one specific clock so you know what's causing instability when an overclock fails.
On the GPU clock, you'll want to increase by increments of about 25MHz. Each time, do a complete run of 3DMark (Ice Storm, Cloud Gate, and Fire Strike). If your overclock is unstable, 3DMark will crash, your system will freeze, and/or your display driver will crash and restart. The only other program I've seen that susses out bad overclocks as well as 3DMark is Far Cry 3, but Far Cry 3 can't be automated and is less repeatable (although infinitely more fun). Once you've found an unstable clock, step back to your last stable test and do a few loops of 3DMark to make sure it's stable. Also, keep track of your 3DMark scores during testing. If for whatever reason you have a score that's lower than a previous run despite your GPU running at a higher clock, your overclock is too high and needs to be dialed back.
Overclocking video memory is much, much trickier. 3DMark can pretty much just be left and automated for the GPU, but video memory will require you to pay attention. I found familiarizing myself with Fire Strike, specifically the second test, helped tremendously. GDDR5 has error correction built into it, so when an overclock is too great, 3DMark is much less apt to crash and will instead demonstrate unusual behavior. You'll want to adjust your overclock in 50MHz increments, and each time, watch your 3DMark run. In Fire Strike's second test, the cinders that arc up through the floor are a really good indicator: if they flicker or strobe, your video memory overclock is unstable. Likewise, if your 3DMark score starts to decline, your video memory overclock is unstable. Step back to an earlier, stabler clock.
Finally, you may see virtually no headroom on your card, or noise may become an issue. Keep an eye on temperatures regardless (HWMonitor is good for this) and if your card is pushing more than 90C, consider raising the fan speed on the card or even going the way of aftermarket cooling.