This sounds like a chilling issue. If a peach tree gets just a little bit too little chilling in the winter, it will button the fruit, which means that it tries to grow them, but they will be deformed and fail to mature. I would check that the variety is appropriate for your area in terms of chilling.
Another possibility is that there is a problem with pollination. If this is the case though, the tree will drop the fruit in late spring or early summer. If you suspect a pollination problem, these articles will help: