First time getting the software set up correctly is a bit of a hassle. This is mainly because the software was originally written for FS98, then hacked to work with FS2000, hacked again to work with FS2002 and now hacked again to work with FS2004 (you get the idea).
Get yourself Squawkbox 2.3 (latest version) and SBRelay (latest version). Also get yourself ServInfo (to see who's online and where).
Install all according to included instructions.
When it says it's ready, connect FS2004 to SBRelay as if to any multiplayer session (this should cause no trouble as it's on your own machine).
Start SBHost (included with SB
to the SBRelay session.
follow the instructions for use of Squawkbox to connect it to VATSIM.
Before doing the above, replace the content of ipaddr.txt in your squawkbox installation directory with this:
which is the current list of VATSIM servers. Most of the ones listed in the original file are no longer available.
For instructions on how to use Squawkbox and VATSIM I point you to the tutorials sections of VATSIM http://www.vatsim.net
and FSNordic http://www.fsnordic.net
which are comprehensive.
You might need to open some ports on your router or firewall to allow all connections.