Protein Language Model Fitness Is a Matter of Preference