I am not following how the bearing is loose. It is a bearing so it should be loose enough to spin easily so the steering doesn't bind but it is contained within the plate that is bolted to the body, so it should not be able to move around.
What cap you are thinking may be worn out? Can you describe the issue a bit more?
Here's a diagram showing the assembly for reference. Perhaps you left a part out?