Certified GKH Gardening Expert
You are on the edge of their hardiness range. Make sure you provide winter protection. They may be suffering some winter damage. Check with the care article below and see if you are providing the optimal care for your trees. Nitrogen is usually the only nutrient needed to apply in spring. Also, they don't typically begin bearing till age 5 so you are getting there.