I think there may be a nutritional or tree maturity component to this. My carrie did the same thing for the first 6 to 8 years that it was in ground. I would get somewhere around 1 fruit per 10 or so panicles -- for a pathetic total of about 30 fruits on a tree nearly 20 feet tall and wide.
As of about 3 years ago, the situation totally changed. I now typically get multiple fruits per panicle. That started around the time I began taking care of the tree with consistent fertilization, irrigation, and fungicidal treatment. So I'm not sure if it's related to tree age or better treatment (or both). I currently have a strong fruit set with many bb-sized fruits forming (at least 1/2 dozen per panicle).
My neighbor's tree, which received no care but which was a few years older than mine has always been a strong producer. So it could be that carrie trees simply need some maturity before pumping out prodigious crops. Or perhaps they get better at extracting limited resources as they age? I did notice that the one year she pruned / thinned her tree it responded by producing a small crop, even though the pruning was not that heavy and was done at the right time. Since pruning the tree would obviously deplete it of nutrition with the loss of foliage, it could still point to this being a nutritional issue.