I think you want to leave them on there until around next June when its warm enough to get vegetative flushes again. If you remove the blooms the tree will use more energy making more flowers. Im not an expert though, maybe the other guys growing mangos in CA have better advice for you.