plums fall off tree
Plums get to about 1 to 2 inches but not ripe and the fall off the tree. I have tasted only 3 ripe plums in 5 years. All the rest are on the ground in various stages of unripeness.
Plums do not ripen all at the same on a tree.
It does sound like your tree could improve from some pruning and fruit thinning.
Here is a few links for you.