The urine could be quite strong. It could eventually build up to level that could harm your trees. I'd be flushing it out periodically. You may want to discourage them from using that area.
Some things others have tried; Laying down chicken wire flat over the ground. Cats like to dig a shallow hole first and this makes it difficult so that they may go elsewhere. Or: If you have or know anyone who has a Liquid Amber tree, the kind who's seed pods are spiky balls, you can collect them and spread them as a mulch. Does the same thing, makes it difficult to dig around the tree and also unpleasant for the cat to walk on.
Repellents: Most good nurserys or garden centers have cat repellents. For the most part they are scent based; various herb oils, garlic, rosemary, etc and egg based, slight rotten egg smell, stronger for a cat's nose than human, but they do smell a bit to us as well.
I believe the toxic aspect is Toxoplasmosis which cat feces may be infected with.
https://en.wikipedia.org/wiki/ToxoplasmosisIt's mostly a problem for pregnant women. No one who is pregnant should ever touch or deal with cat feces, especially feral cats. It won't be absorbed by your fruit, but the soil could have levels in it that would not be wise to have around. That right there is a good reason to discourage the cats from using your trees as cat box.
After trying the above methods to get them to leave your tree area, if you still want them around to take care of rodents you could try to lure them away from that area to another less problematic area by putting down sand or fine gravel in a better spot, so they hopefully use that area instead of your trees soil.