Certified GKH Gardening Expert
This is very common, and it is not a cause for concern unless you don't get any ripened fruit at all. This is when you know you have an issue. Otherwise, this is done because most of the fruit is lost to pests, anyway. Years of adaptation has led the tree to overbear, so that it has a better chance of ripening a few fruit for reproduction.
Now if you don't get any fruit at all, you will want to pinpoint whether it is pests or just not ripening at all.
This article will help: https://www.gardeningknowhow.com/ornamental/trees/fruitless-mulberry/mulberry-fruit-drop.htm