Certified GKH Gardening Expert
There are some brown rot-resistant peach varieties, but I'm not sure about plums. Brown rot is not believed to overwinter in the soil, so if you can manage to remove every bit of fallen fruit you should be fine. Treating the ground with fungicide would be a good idea just to make sure.
These articles have some information on management and fungicide options, but the available products vary by state.