Q.plums fall off tree
Plums get to about 1 to 2 inches but not ripe and the fall off the tree. I have tasted only 3 ripe plums in 5 years. All the rest are on the ground in various stages of unripeness.
Plums do not ripen all at the same on a tree.
It does sound like your tree could improve from some pruning and fruit thinning.
Here is a few links for you.
https://www.gardeningknowhow.com/edible/fruits/plum/picking-plum-fruit.htm
https://www.gardeningknowhow.com/edible/fruits/plum/plum-fruit-thinning.htm
https://www.gardeningknowhow.com/edible/fruits/plum/plum-tree-diseases.htm