Small worms that I also find falling off trees into my pool, very strange looking - not smooth skinned but prickly looking.
Small Black Worms Eating Flowers
Certified GKH Gardening Expert
More likely a caterpillar then a worm.
Some caterpillars can be eliminated with something like Neem Oil.
Correctly identifying the infestation will make the treatment more effective.
Here are some links for you.