I once told a guy I work with "I don't speak English" and he just said "oh, okay" and walked off. He hasn't bothered me since.