It will not hurt the plant to cut these off. You can remove them at any time, unless you want to use the seeds. Then you should wait until the seed pods are dry before harvesting them.