Certified GKH Gardening Expert
You may have to hand pollinate the female buds. sometimes the males and females show at different times, so if you cannot find one to pollinate with, then wait for one and dry the flower with pollen in it for later use.
Sometimes this can also happen from a lack of nutrients.
this article will give you more information: https://www.gardeningknowhow.com/plant-problems/environmental/why-healthy-blooms-fall.htm