Certified GKH Gardening Expert
It is normal for limes (and occasionally lemons) to drop their "pea size" fruits just after flowering in spring. They may also drop them again in summer due to the warmer temps. Excessive dropping could be attributed to stress, however.
Here is an article that you may find helpful: https://www.gardeningknowhow.com/edible/fruits/lime/is-lime-fruit-and-lime-blossoms-falling-off-tree-normal.htm