Does the damage look like it was done by larger animals like mammals or birds, or by insects? Check on and in the fruit for any insects or larvae. Check this article for types of insect pests and what to do about them:
If the culprit seems to be birds or mammals, you can surround the tree with netting designed to keep birds and mammals off once the fruit begins to grow. Check under the netting every day in case an animal manages to get inside.