The tree bore very little fruit this year and is in need of a thorough pruning. There is no sign of any pests on or under the leaves.
My Plum Tree Has Developed Small Holes in Its Leaves
Certified GKH Gardening Expert
It depends on what the holes in the leaves look like as to what it may be.
If the holes look like they have been chewed out, it could be slugs. They have been known to get up in trees and eat holes into leaves at night - which is why you would not have seen them.
If the holes are ringed with yellow, it may be a fungus. In this case, treat the tree with a fungicide.