Certified GKH Gardening Expert
It depends on what the holes in the leaves look like as to what it may be.
If the holes look like they have been chewed out, it could be slugs. They have been known to get up in trees and eat holes into leaves at night - which is why you would not have seen them.
If the holes are ringed with yellow, it may be a fungus. In this case, treat the tree with a fungicide.