If Southern California and Florida have shown us anything, it is that large numbers of people tend not to equal large numbers of citrus. It tends to equal tract housing, fast food restaurants, and many, many parking lots.
So yes, I do believe that a remote, sparsely populated island in the South Pacific could potentially have the world's largest citrus collection. Ua Huka could be like New Zealand. In New Zealand, sheep outnumber people. On Ua Huka, citrus varieties could outnumber people. Remember, all it takes is one person, a small group of people, or an organization and a large enough tract of land to create an outstanding fruit collection. Add in a naturally disease-free location and time for the collection to accrue plants and mature, and you can have a very spectacular collection.
In Ua Huka's case, you have Léon Litchlé, Mayor of Ua Huka and Marquesan Minister for Agriculture, who founded the arboretum. Later, the French organization CIRAD and specifically citrus expert Alain Sizaret would take part in the development of Arboretum Papua Keikaha's citrus collection. The arboretum sits on 17 hectares (42 acres), and the citrus collection was allocated 2.5 hectares (a little over 6 acres) in 1995. The arboretum has had nearly thirty years to accumulate citrus varieties, with help from INRA. There is no greening in the Marquesas Islands to kill off the collection, either.
Now, does Arboretum Papua Keikaha definitively have the largest citrus variety collection in the world, larger than the INRA-CIRAD Citrus Germplasm Collection in San Giuliano, Corsica or the UC Riverside Citrus Variety Collection. Possibly, but likely not. Despite what Tahiti Tourisme states on their website, Arboretum Papua Keikaha does not claim to have the world's largest citrus collection in their 2010 guidebook and "only" claims to have several hundred varieties. INRA and UCR both claim to have around 1,100 varieties. In INRA's case, it claims "more than" 1,100 varieties, and in UCR's case, "nearly" 1,100 varieties.
However, what Arboretum Papua Keikaha may rightly be able to claim is the largest citrus collection in the world that is open to the public. Unless I am mistaken, INRA and UCR are not open to the public, since they are research institutions that do not want people potentially exposing their collections to plant pathogens. Plus, they don't have the staff to deal with many visitors; the staff they have are very busy doing research and trying to keep their collections disease-free.
Either way, whether this arboretum is the largest citrus collection in the world or "just" a very large citrus collection, if I ever find myself adrift at sea in French Polynesia, I now know what island to set sail towards.