These cars are getting very old now.
You have five axles and U joints and five motor mounts connecting everything together.
All of them while possibly still all fine they're all starting to wear or perhaps one is failing.
Most of these cars do a little clunk or clank now when shifting back and forth.
As long as it doesn't become much worse deal with it.
It always helps to shift slowly and stop in neutral for a split second each time you go back and forth.
If you're concerned, have someone who knows what to look at, check your lower motor mounts, especially the rear (tranny) mount.
It may look "OK" visually, but if you have over 100K miles I can almost guarantee you that it needs replacement and probably the front (lower right) mount too