Certified GKH Gardening Expert
No, you don't have to, but some people choose to do so only so that the plant focuses its energy on leaf and root growth during the first year or two. It's really a personal choice. If you leave them, it is likely you will see some fruiting this year and, if not, definitely by next year. They reach full maturity by year 3 or 4 and you will see far more fruit on the plant than you did the first 2 years.