Certified GKH Gardening Expert
I wouldn't, necessarily, take off any leaves. These plants tend to be a little on the finicky side, and it is best to cause as little stress as possible.
Now, you can carefully cut spent blooms. This will be fine. This article will help: https://www.gardeningknowhow.com/houseplants/african-violet/african-violet-care.htm