Literally every single point-and-click I've played has extraordinarily arbitrary puzzles, so that rule might not apply.
Viewing post in What is an Adventure Game?
An example of an arbitrary vs a non-arbitrary puzzle.
A non arbitrary puzzle would be where you have to get some wood, nails and a hammer, and build a ladder to get something out of a tree. There are also other things that you have to do to get those items. Like solving a dispute with the lumber company so that they can supply you wood, etc etc
An arbitrary puzzle is you have to complete a crossword puzzle and then for some weird reason that allows you to get into the tree.
A few arbitrary puzzles can be fine in an adventure game. But if the game you're playing has just a bunch of arbitrary puzzles, it's probably a puzzle game, not an adventure game.