I would say I have this difficulty myself. It's mostly the visual overload of the bright lights in an otherwise very dark environment. That, and being awake all day has generally depleted the emotional/neurological reserves that help keep us centered. We've been going hard at it all day, so by the time night comes, we're worn out to begin with.

I have started tilting my rearview mirror to utilize the 'night driving' position, which does tame the lights behind me, though oncoming traffic remains an issue. Some people wear sunglasses, even at night, but sunglasses are really harsh on my tactile system, so they're a no-go for me, hehe.