If you're a surfer, the best time would be December period when the waves are super high and the surfing community set up their tents for the duration.
Try not to go during the festive season if you can. Or otherwise book way in advance.
Accommodation and Packages to Pahang ~ Cameron Highlands, Taman Negara, Kenong Rimba Reserve, Fraser's Hill, Cherating, Pulau Tioman, Kuantan, Tasik Bera and Surroundings