Certified GKH Gardening Expert
These sound like fungus gnats. They are not harmful, and in fact, can be beneficial to the soil. They will really only come out when the soil remains too wet for too long.
I would make sure that the soil dries out pretty good between waterings to prevent these from being overbearing. They do contribute to healthy soil, but they can be a problem when the soil is too wet.
If you still want to get rid of them... Rosemary oil in warm water left to cool then watered in on the next watering will rid the soil of these. Just be aware that I do not recommend totally ridding the plant of them. They do help the soil out quite a bit.