First, run ssh-add -K and check whether this fixes your problem.

If not:

• Removed the rsa_id.pub file and regenerated a new one (must be in ~/.ssh/):
  ssh-keygen -y -f id_rsa > id_rsa.pub
• Ensured permissions were set to 600 for both id_rsa and id_rsa.pub (must be in ~/.ssh/):
  chmod 600 id_rsa*
• Ran the following command:
  ssh-add -K

After doing this, I was no longer prompted to give my private key password. This appears to actually put the private key password in the correct keychain location for OS X to use.

source: http://apple.stackexchange.com/questions/18458/password-dialog-appears-when-ssh-private-key-permissions-are-set-to-0600