Certified GKH Gardening Expert
It sounds like a pollination problem. If the flowers are not properly pollinated, the fruit will be small or deformed. This article will help you with understanding about pollinators:
Hand pollinating will also help. Simply take a small paintbrush and swirl it gently on the inside of a blossom, then move on to a new blossom and swirl the brush in the inside of that blossom. Repeat with all the flowers.