Certified GKH Gardening Expert
It is possible that your tree is aborting its fruit due to an over abundance of fruiting. This is normal and the reason why thinning is often performed on fruit trees. Issues with pollination may also be to blame.
For additional causes, here is an article that you may find helpful: https://www.gardeningknowhow.com/edible/fruits/lime/is-lime-fruit-and-lime-blossoms-falling-off-tree-normal.htm