That's working as intended, since you're not really meant to move forwards at speed in hover mode, and not really meant to hover up/down in the manual mode. However I am interested in what you were trying to do that you felt you needed to do either of those things in those modes.