Is the Milkweed invasive? Does it grow with runners?
If my comments about milkweed left you hesitant about planting it, consider butterfly weed, Aesclepias tuberosa. Like all milkweeds, it is drought tolerant. More importantly, it spreads less and is far less likely to send up shoots a distance from the main plant.
Common milkweed, Aesclepias syriaca is a thug in good garden soil. It spreads via underground runners in every direction. When growing milkweed, place it where the soil is not the best and look into which ones are native in your state since the butterflies in that area are used to certain varieties. (There are over 100 types of milkweed in the US) Swamp milkweed and Sullivant's milkweed are said to be better behaved in a traditional garden. Or consider having a bed just for pollinators and butterflies and let the milkweed spread. https://www.wildones.org/wp-content/uploads/2013/06/Meaningful-Gardening.pdf