Certified GKH Gardening Expert
It may be that your are waiting too long to harvest your limes. The fruit should be harvested before it turns yellow, limes are picked before they are ripe.
Make sure the citrus trees are also receiving adequate water.
Here are a few links for you to read through.