Certified GKH Gardening Expert
Wash the plant down with soap water (made with a dish soap that does not contain bleach). This will help get rid of the oil. The leaves may already be a lost cause, but if you care for the plant as you normally would, the leaves should regrow rather quickly.
In the future, always test solutions you mix on a small part of the plant first and let it sit for 24 hours, just in case something goes wrong.