Certified GKH Gardening Expert
This has two likely causes. The first would be underfertilization. I would make sure to apply a fruit tree fertilizer once per year, or a nitrogen based in spring, and switch to a high phosphorus and potassium fertilizer once the fruit start to form.
The second would be a fungal infection. This is common during wet seasons. Wettable sulfur, and dolomitic lime should be a once a year application, just to be sure, anyway. Add these if you, also, suspect a fungal infection.
This article will give you more information on the care of plum trees: https://www.gardeningknowhow.com/edible/fruits/plum/growing-plum-trees.htm