Perhaps it has exhausted the ground of the nutrients it needs and this is why it hasn't been so productive over the last 4 years. It may also explain why the other tree, just a short space away, is also ailing.
Try giving your mulberry trees a good, general purpose fertilizer such as blood and bone in the late winter. Apply at a rate of 2 oz a square yard (70 g/square meter). Dig it lightly into the soil all around the tree. Remember that the extent of the roots will go to the edge of the tree's canopy, so place the fertilizer right to the end of the root line.
In the following spring you can apply a mulch of well-rotted manure around the tree, making sure that you do not place it too close to the trunk, as this can cause disease.