The triangular panel does come off, but you can't pull the speaker from behind there. You will have to remove the entire door panel, but it's not like it's that much work.
2 covered screws at the front, 5 screws along the bottom, 2 covered screws in the door handle, 1 under the latch, 2 under...