My dwarf Washington navel tree sets a decent amount of fruit, but most are inedible at winter ripening due to very small size. They are just a bit larger than golf balls on average.
Orange Tree Problem
Certified GKH Gardening Expert
Drought can be a cause of small fruit on citrus trees. If that's not the issue, thinning the fruit might help. Most citrus trees will self-thin by dropping a significant number of fruit, but if yours isn't doing this enough, you can remove most of the small fruit when they've just begun to develop after pollination. This should help the remaining fruit grow larger.