Certified GKH Gardening Expert
This can be caused by a few things.
Moving from inside to outside is quite stressful to the plant. Especially if you have to do this several times per week. It will be natural for bud drop to occur during this time.
Sometimes, adding phosphorus and potassium can help prolong buds, but if environment is against you, then you may have to wait a few years longer than normal for the plant to cope.
Alternatively, growing it indoors year round under horticultural lighting can grant you fruit rather quickly. It will be a lot better for the plant to remain in a steady environment and will provide better conditions for flowering.
This article will help you to grow the plant properly: https://www.gardeningknowhow.com/ornamental/vines/passion-flower/passion-fruit-vines-in-pots.htm