He doesn't hate being petted on the hat I think he don't want you to know hes not really human thats why he hinch and held his breath when you touch his hat it's how I take it cause he likes it when you touch his head cause he rapidly blush after u pet him but if you don't pet him he has a gentle blush (this is only head Cannon)