I mix my own ferts. I have a massive chemical collection left over from when I did hydroponics, also from scratch.
You can do the same without too much difficulty if you only want to focus on the highest-in-demand macros. The main ones are NPK, and you don't want N, so P and K. The next most in-demand are Ca, Mg, and S. So, for example, you could use superphosphate for calcium and phosphorus, epsom salt for magnesum and sulfur, and potassium sulfate for potassium and more sulfur. Just a random possibility. Potassium chloride (salt substitute, both for eating and for driveways) may be cheaper and/or more available, but you don't want too much chlorine in the soil (some is needed, but your plants should have no trouble getting enough on their own). There's lots of possibilities. Basically, all that matters is that what you're adding are soluble salts of minerals that you want. Once they dissociate in water, it doesn't matter what form they came in.
Of course, you can also buy low-nitrogen ferts pre-mixed.
Look for "blooming" or "fruiting" mixes.
Whatever you do don't stop fertilizing completely. Flowers and fruits take a lot of nutrients in their own right. Nitrogen, however, is used primarily for making leaves.
Hah, most people hate the fact that caterpillars eat up their passionfruit