Personally I wouldn't really worry about the flowers, just make sure you don't hold too much fruit on a young tree. There's a big difference though between a 4' single whip atemoya and a 4' atemoya with a ton of branching, but either way I wouldn't let it hold more than a fruit or two if you're determined to let it hold some. If you pulled that many flowers off of your tree I'd guess that is has some decent branching to it and is likely a healthy tree. A pic of your tree would help.