I meant to respond to this earlier, but I lost track of it.
Here's what you need to do to remove the door panel. It's a lot of little steps, but really it's simple.
Unscrew the lock knob at the top of the door.
Roll the window all the way down.
Pop out the trim around the lever that opens the door.
Pop out the 2 plugs from the armrest and remove the screws (3). Now the armrest is only held on by the post at the top. Pop out the power window switch panel. Disconnect the switches from the wiring and mark the wiring and/or switches if necessary. Rotate the armrest about 90 degrees and pull gently at the same time. There's a peg that has to line up with a slot in the door for it to come out. Once it's out, you'll see how it works.
Remove the plastic screws from the bottom of the map pocket, if there are any remaining.
To remove the pocket, grab one side right around where the screw is and pull straight up slowly and carefully to slide it off the plastic spacer. Same with the other end.
Grab the bottom of the pocket and pivot it up and out from the door, and the center tab will come out.
Remove the two screws that held the door pocket on.
Work around the edge of the door panel (bottom first, then sides), prying it away from the door. There are little plastic plugs that snap into the door, and some will break while removing the panel. They aren't too important, so don't worry about replacing the broken ones.
With the two sides and the bottom loose, lift the door panel up parallel to the door and it will be off.
Disconnect the wires to the speaker, marking which wire goes to which terminal, and now it should be free from the door.
Installation is reverse of removal. If you look at the top of the door panel on the back side, you'll see that there are two wide metal tabs that hook on the top of the door. I've found that it's easier to get one of those on at a time, starting closest to the mirror. I usually get that side started first, then slide it over towards the mirror and then drop the panel over the threaded rod (for the lock mechanism) and it falls into place much easier.