Oh yeah...thank for mentioning that. I forgot all about that since I started to make it so you couldn't collide with him, but then stopped cause it wasn't adding to the MVP. I'll add this to my backlog so I don't forget to fix it later.
As a side note the same thing can happen for the enemies, but I think it's less noticeable because you are trying to actively get them off of you