Looks attract, but personality keeps.
In short, your looks can get you talking to them, but your personality will get them "annoyed" as you put it. You are probably coming across too much as a guy who is a pig, or too into himself. You have to make it about them, and be genuinely interested...