your family vs. my family....DING!

We will have been married 7 years on Monday. We don't fight that often but we have a couple fights where I feel like I'm talking to a brick wall. This is how one of our usuals goes:
P-My parents called to see if we'd like to come down and _____.
C-Oh man, I'd love to, I just hate making the drive.
(For reference, it's 2 hours down and 2 hours back.)
P-If it was YOUR family we'd go.
C-My family lived 2 miles away.
P-You hate my family.

Ok...how in the world do we get from "it's so far" to "i hate your family"???? Can someone help me connect this?!?
And it's the SAME discussion every time. Do I get ANY points for initiating the birthday lunch for his niece last weekend? no. Do I get ANY points for having the kids over EVERY halloween weekend to do fun stuff-always involving WAY too much sugar? no. Do I get ANY points for always being the one to initiate holidays, gifts, birthdays, easter sundays together? no. I only get demerits for not wanting to drive two hours. *ugh* I could pull my hair out!!!