Something eating blooms
Something is eating the new blooms when they first start growing? I find the brand new blooms eaten off clean. All of them.
Certified GKH Gardening Expert
Difficult to determine from your description.
I would advise a Neem Oil treatment, as this works as both an insecticide and fungicide.