Q.Why Are My Ginkgo Leaves Yellow In July But Not Falling Off
I have planted spider lilies, a rosemary bush, and native milkweed. I live in Richmond, VA. My tree is male and about 10 years old and was propagated by an local arborist at the Lewis Ginter Botanical Gardens. When I just had spider lilies around it, it seemed fine. This year has been very hot with some rain about every other week. The tree is about 20 feet tall and has a lot of weeds and vines around the base because after I planted the milkweed, it became too difficult to keep other weeds from coming up. The milkweed has really taken off.

Certified GKH Gardening Expert
There likely is too much competition for moisture and nutrients. I suggest moving the butterfly weed somewhere else. You may want to fertilize it, too.
https://www.gardeningknowhow.com/ornamental/trees/ginkgo/feeding-ginkgo-trees.htm