Certified GKH Gardening Expert
I had the same problem in my butterfly garden. If it wasn't the birds eating the caterpillars, it was the wasps. About the only thing you can do to protect the caterpillars is to cover the plant with netting once you have several caterpillars. The other option is rearing them in butterfly rearing cages so you can protect a few. The birds like to feed the caterpillars to their young, and once they find you have a good supply, they will stick around.
Here is an article on discouraging nuisance birds: