How can I avoid this person (family member) forever without ever really having to explain to him why?

I'd say you should see a psychiatrist, but a good one. I saw one for social anxiety and it didn't help much. Pretty much I just tried to force to negative thoughts and anxiety away whenever they came up.