As mentioned, X is essentially a networking protocol with graphical displaying capabilities. This makes for some interesting usage possibilities. And also means there are inherent security considerations, as there is with any networking environment. And if you ever connect to the Internet, you are in the midst of one very large, hostile network ;-)
X clients connect to X servers via various networking protocols, including TCP/IP. Even with just local connections. Possible usages here are to run an application on one computer, and display it on another. Or, to actually log in to a remote system, and have it display to your local screen, with the client apps using the remote system's CPU and RAM.
Without any precautions, this can leave you wide open to various types of mischief and abuse. For instance, anyone logged into to your system can access your "display", meaning they can see what you are doing if they want to. Thankfully, most recent Linux releases come with some default security precautions enabled. But it is best to make sure for yourself that you are protected.
Both X networking and security are nicely covered in The Remote X Apps Mini HOWTO, http://www.linuxdoc.org/HOWTO/mini/Remote-X-Apps.html, so we won't need to try to rehash it here. Recommended reading. See other references in the Links section of the Appendix below.
A few recommended precautions:
Never, ever run X as root. The number of bad things that can happen, dramatically increases when logged in as root. Learn to run as much as possible as a regular user, and su to root only when needed. This may sound like a lot of extra work (and probably is at first), but once the "right" way of doing things is learned, it soon becomes second nature.
A brief anecdote from a friend: he had a client who's new system stopped "working". Curiously, he found the entire /dev directory was missing, which he re-installed and all was well again. He was back a few days later and found the system logged in as root to X, and someone had clicked on /dev in the file manager, and dragged it onto the desktop. Smooth move!
If you ever connect to a network with untrusted users, be sure to have a firewall between you and them. This goes double for the Internet. Firewalling is beyond the scope of this document, but is covered in many other places, including your vendor's website. http://linuxdoc.org has several security HOWTOs that can help as well. http://linuxsecurity.com/docs/ is another good place to look.
You can disable TCP connections with the "-nolisten tcp" command line X server switch. This does not help for local connections though. For xinit/startx:
exec X :0 -dpi 100 -nolisten tcp
Placed in ~/.xserverrc. And for xdm, in /usr/lib/X11/xdm/Xservers:
:0 local /usr/X11R6/bin/X :0 -nolisten tcp