The leaf damage looks like mite or thrips damage in cold weather, but the sick branches dying back looks to be some vascular infection of bacteria or fungus. Put some still-alive branch and leaves into a paper bag, not plastic, and mail to TREC. Tropical Research and Education Center near Homestead, Florida.
Meanwhile apply some Alliette or Flanker, or some phosphite product (such as Elemax Foliar Phosphite, or Organocide Plant Doctor.