It is several years old. Each fall we trim it down to its base. In the spring, many little "would be" figs appear, but fall off the tree shortly thereafter. We have not fertilized it ever. It is planted on the east side of our house and gets full sun for the most part. What are we doing wrong?