I kind of feel the slowness of the movement is actually intended. After all, a boss doesn't move quicker because you died a lot in a real videogame.
