Certified GKH Gardening Expert
You could be overwatering and you could be underwatering, or the plant could be suffering some other stress as well. The yellowing leaves and dropping fruit are a stress response by the tree.
If you are overwatering, the tree will be in standing water. You should improve the drainage or hold back some on watering. If you are underwatering, you will notice droopy leaves frequently on the tree and the soil will be dry frequently.
If neither of these is the issue, the tree may be lacking in nutrients and need fertilizer or it may have a pest or fungus problem and need to be treated with a pesticide or a fungicide.