In my early days, before I understood that my Hailey's Comet was self sterile and I was paying attention, I found worms too in the flower remains. With a google search, I found some farmers in some far away place complaining that fruit flies were destroying their DF crops and I thought that was my problem. If there is a lot of rain, the flowers turn to slop and maybe that makes it a good breeding ground. If there is no rain, they just dry up. Either way, I let them fall off by themselves and sometimes I harvest the fruit with the dried up flower still attached. This prevents ants from farming in the front cavity.
Now that I pollenate with a different DF variety, I rarely ever have a flower not produce a fruit. In 1 in 25 fruit, ants break through the skin or breaks from too much rain, exposing the flesh. I remove the exposed area, even though it still apears ok, just before eating. I have only found worms in fruit once and it was very evident from the outside that things had gone bad. The skin was broken and black in an area. I cut away that quarter and the rest of the fruit seemed fine. I have never found evidence of an animal eating a DF either.
