Taskmanager is only showing the D3D Usage, not the real core usage. Download HWinfo64 to see the actual core usage.
Regarding your VRAM, it uses the VRAM proportional to your GPU-Performance.
The only other thing that's impacting your VRAM usage is the Video-Resolution.