Tapered optical fibers (TOFs) are used in many areas, from the coupling of light onto optical chips to optical sensing and amplification to interfacing quantum emitters. Using laser light, we study the fundamental torsional mechanical mode of the nanofiber-waist of a TOF. We find that this oscillator features a quality factor of up to 10 million and a Qf product of 1 THz. We damp the thermal motion to temperatures as low as 28(7) mK by means of active feedback. Our results might enable a new type of a fiber-based sensor and lay the foundation for a novel hybrid quantum optomechanical platform.