Certified GKH Gardening Expert
It sounds like a watering or nutrient issue. Increase watering a bit and give it some fertilizer higher in phosphorus. This will help with fruit production, which saps the tree of energy. Just to be safe, you should also check for any signs of pests. If you find any (or signs of their presence), you can treat them with neem oil, which is also an effective fungicide (should anything else be the issue).