I got around that by saying I'd work to do in the shed for him and that to do it he'd need to give me the key for the shed. I'd just repeat that he needed to give me the key to the shed before I could leave until I got the key.
