By design, Full Contact™ will not stop running if you close the window. It continues running in the background until you open it from the system tray again. To shut it down you need to use the Exit menu item in the right-click menu on the system tray icon.
At that point the program will exit and the system tray icon will disappear.
