Certified GKH Gardening Expert
It may be a pollination issue, which, if the case, is not too late to fix. The limes will only grow if the flowers were properly pollinated. If they were not properly pollinated, they will stay small and eventually fall off the plant.
These articles may help for the future: