Thank you so much for playing!
Yes, both the amount and color of the bait matters. More red bait increases the chance for rarer red fish (decreasing the chances for lower star fish!), same for the other colors. There's even a special combination, you'll notice that you have the correct one if the frame changes how it looks!
So it seems like you were mostly unlucky, unfortunately :')
What fishes were you missing in the book?