Yesterday, I set out to enable key authentication on all of my Linux servers. Once I had exported my private and public keys from my putty key file, it was a pretty simple process.
Bar, one server. IT worked flawlessly on all but on of my servers. This was a mirror of a Fedora server I had at home, so decided to copy all of my .ssh and /etc/ssh folder across, hoping a quick restart of SSHD would fix the job.
Well spent hours last night thinking what could it be. AH, I remembered ssh-add was fussy about the folders permissions, of where your key was kept. Apprarnetly ssh is the same about your .ssh folder. A quick google on this confirmed this fact, and issuing the below commands fixed all 
Typically this is caused by the file permissions on $HOME, $HOME/.ssh or $HOME/.ssh/authorized_keys being more permissive than sshd allows by default.
In this case, it can be solved by executing the following on the server.
$ chmod go-w $HOME $HOME/.ssh
$ chmod 600 $HOME/.ssh/authorized_keys