My lime tree (planted in a large clay container on the patio) had a large crop of blooms followed by the onset of fruit, which is about pea size now. For the past week the fruit has been dropping off slowly. I try to keep it evenly watered, not letting it dry out. It was fertilized about one month ago. I think it is because it is a rather small tree (2 ft in height) and it has produced too much fruit for the tree to support.
Lime Tree Dropping Fruit
Certified GKH Gardening Expert
It is possible that your tree is aborting its fruit due to an over abundance of fruiting. This is normal and the reason why thinning is often performed on fruit trees. Issues with pollination may also be to blame. For additional causes, here is an article that you may find helpful: https://www.gardeningknowhow.com/edible/fruits/lime/is-lime-fruit-and-lime-blossoms-falling-off-tree-normal.htm