After closing out a character’s profile, the panel returns to your own profile (This just might be the default). It might make a better UX flow to have the panel collapse.
As it is currently, every time you look at someone and close their profile, you then have to manually collapse the panel.
A good solution might be to remember whether the panel was open prior to looking at someone, and then collapse it again once the profile is closed.
I see what you mean. And the fact that you bring it up means it could be made more intuitive.
Yes, the behavior is much like you explain. The panels work like stacks of pages, where each panel has a default “bottom” page:
When you click something that opens a different page, it will be added to (or replace) that panel’s stack. Clicking X will close the top-most page. And once you reach the “bottom” of the stack (a.k.a. the panel’s default page), the X will disappear.
But I like your idea! 
After all, we have the feature of the panel being automatically shown when looking at someone. So it really should work in the opposite direction too; hiding when you stop looking.