Yes, you have the right idea. Pull them off when they get pea size. If it blooms again, repeat the process. It is warm enough now that I doubt your trees will bloom a third time. I've had more experience with lychee, but I think the same principle applies.