Certified GKH Gardening Expert
This can happen for a variety of reasons. The most common is that the tree is getting too little water and it will abort any fruit in order to keep its water usage to a minimum. This article may help with that:
Another common reason is that the flowers were not pollinated. This can be very common in potted trees that may not have access to pollinators. You may need to hand pollinate. This article will help with that:
Another reason might be just natural shed. A tree will naturally get rid of some of its fruit if it feels that it cannot support.