We report on the different roles of two orbital-active Fe2+ at the A site and V3+ at the B site in the magnetic excitations and on the anomalous spin-wave broadening in FeV2O4. FeV2O4 exhibits three structural transitions and successive paramagnetic (PM)-collinear ferrimagnetic (CFI)-noncollinear ferrimagnetic (NCFI)/ferroelectric transitions. The high-temperature tetragonal/PM-orthorhombic/CFI transition is accompanied by the appearance of a large energy gap in the magnetic excitations due to strong spin-orbit-coupling- induced anisotropy at the Fe2+ site. While there is no measurable increase in the energy gap from the orbital ordering of V3+ at the orthorhombic/CFI- tetragonal/NCFI transition, anomalous spin-wave broadening is observed in the orthorhombic/CFI state due to V3+ spin fluctuations at the B site. The spin-wave broadening is also observed at the zone boundary without softening in the NCFI/ferroelectric phase, which is discussed in terms of magnon-phonon coupling. Our study also indicates that the Fe2+ spins without the frustration at the A site may not play an important role in inducing ferroelectricity in the tetragonal/NCFI phase of FeV2O4.