Sometimes I have the same problem with the warm-water species. They will try to take the bait but are unable and will flip back and forth 180* around the bait as a pivot point. If I'm not watching it happen, they're able to take it. It tends to happen when I'm looking directly down the hole.
It also happens at times in underwater camera mode.