Don't worry too much about the 'matching', despite what your neighbor thinks.
Most of the time, speakers are killed by not enough power, causing clipping, which makes for uncontrolled movement of the speaker cone, which will tear it apart.
Rarely, you can push too much power through a...