It would make a difference, the attacker would just go to the site name, look for the ip however he does that, and easily get your new ip. Hmm why would you think never?
The attacker wants to attack many IP addresses right? I think once he finds out about the new ip, he could be back very soon.