Worried about my heart?

I was just wondering if congenital heart defects can be hereditary? My Grandad died of a congenital heart defect when he was 23 (Just before my dad was born) and my heart never feels right. I've been to the doctors and they said it was panic attacks i was having. But i know its not. Its just irregular heart beats.