A question about potential phobias?

Well let's first define a phobia, which is an anxiety disorder and fear of someone or something. Hating when people whisper around you is not a phobia and neither is hating people with bad manners. However, if you have a phobia of people going away while talking then I suppose that could be some form of Isolophobia, which is a fear of being alone, that is, if that's what you meant. To me, it sounds like you do have one phobia that I can confirm, and that's phobophobia, a fear of phobias.