Certified GKH Gardening Expert
Being in zone 4, there is a possibility that frost is killing the blossoms early in the spring, before they have a chance to open. A light frost will kill the blossoms but leave the foliage undamaged.
Start checking the tree early to see if you can spot flower buds and, if possible, try to cover the tree with a sheet if any frost is expected to try and save the buds.
Also, double check that the peach tree you have is appropriate for zone 4. There are only a few that can grow and bloom in your zone.