Q.Why do my peas die off after one picking (4 or 5 pea pods)
Leaves are turning brown already

Certified GKH Gardening Expert
It would be hard to say for certain without seeing the damage. I can take a few guesses and give you information on curing common issues with them, however.
Most likely, the symptoms are describing a fungal infection. This will need to be treated with a fungicide. Most of the time, this occurs from the soil remaining too wet for too long without a drying out between watering.
Less likely would be a combination of deficiencies such as iron, phosphorus, and potassium.
Here are a few articles that will help you:
https://www.gardeningknowhow.com/garden-how-to/info/using-fungicides-in-garden.htm
https://www.gardeningknowhow.com/edible/vegetables/peas/pea-plant-diseases-and-pests.htm
https://www.gardeningknowhow.com/edible/vegetables/peas/how-grow-peas.htm