Certified GKH Gardening Expert
This is known as "fruit drop". Some amount of fruit drop is normal in citrus trees, but excessive fruit drop can be caused by extreme temperatures, swings in temperature, watering problems, or other environmental challenges.
See the "fruit drop" section of this article for more information:
This article is about lemon fruit drop, but the advice applies to grapefruit as well: