Given where you live, I would suspect that they are not getting enough water regularly. Trees will often abort fruit first when under stress and then may drop leaves, without yellowing, if the water shortage is severe. It is the tree's way of saving itself.
I would recommend making sure the plant has plenty of water consistently. If you water regularly now, I would recommend either increasing the amount of water or the frequency of watering.