Try to get out on a boat one evening for some squid fishing yourself. Make a visit to Phu Quoc National Park and swim in a quiet pool.
Motorcycles are plentiful and cheap, hire one off a local and go cruising around the island. Petrol can usually be purchased around the island at various beaches served fresh in glass bottles.
Also Bicycles is a great way to observe the island. You can rent a mountain bike for around 50,000 VND per day at Saigon Phu Quoc Hotel or some local shops (be sure to write ‘xe dap’ because few Vietnameses here could speak English well).
Some beaches stretch for kilometres with overhanging palms, no one else in sight what more could you want?
Try SCUBA diving. The shallow waters (max deep to 12-16m) allow even unexperienced divers to have nice dives. There is some nice coral to see and if you are lucky you might see a bamboo shark.
Also, watching the sunset at Pho Bo Cafe, a 2 storey place with the best view overlooking the lighthouse and the shrine, is the most romantic activity for couples and artist.
Later, try wandering around the night market (near Ferry landing in Duong Dong Town), where locals mingled with the tourists, a place that offered a selection of goods, pearls, seafoods, vegeterians, or even a playground for children.