Well, that's a ridiculous reason to be impolite to someone else. I'm sure it must be quite unsettling to meet someone that looks exactly like you, but that's still no reason to forget manners.

I suppose it's because it's hard to really accept that time isn't moving there, when it is here. And if someone was in the middle of something very important when they left, and now all this time they have to sit here thinking about it and feeling guilty over leaving in the middle...I can see how that might bother someone.