Do you have functional male and female trees? Males must be available to provide the pollen for pollination.
In some cases, pollination may occur, but because of weather conditions or nutrient issues, fertilization may not, resulting in shriveling of the ovule rather than fruit development. Pollination can stimulate fruit set, but then the nuts fail to fill.
Nutrition can play a role; boron deficiency can be a factor in some areas.
A soil test can help you determine if this is the issue.
Weather temps can also effect nut growth.
Drought stress can also effect nut growth.