Q.Vitex
I planted some ferns at the base of my vitex plants (4) and you know how ferns are. Is it possible that with so many ferns the soil is becoming too organic? The vitex are just starting to leaf and it doesn’t look right.

I would inspect your Chaste Tree for any other signs of issues. Look for pests or leaf damage.
If the soil is holding too much moisture with the underplanting, this could very well be an issue for your tree.
Check the soil by digging down a few inches. If is seems excessively wet, your tree may not be happy and over the long term could kill the tree.
https://www.gardeningknowhow.com/ornamental/trees/chaste-tree/chaste-tree-cultivation.htm