Best work around would be to make enemies still collide with each other, however make the player invisible to the enemies and then keeping all the other collisions
Damn, it's really easy, thank you!