Can someone else interact with the panel once the glass-wearing user is gone?

No, because the UI and the interaction are happening within the first user's glasses, and once he's gone the panel is not being seen by any glasses and therefore cannot be interacted with. Remember that the physical panel is a purely passive surface that is only there to show the glasses where to overlay a UI; it contains no interaction capabilities in itself such as touch detection.