Often, something will not be quite right when you initially fire up the X server. This is almost always caused by a problem in your XF86Config file. Usually, the monitor timing values are off, or the video card dot clocks set incorrectly. If your display seems to roll, or the edges are fuzzy, this is a clear indication that the monitor timing values or dot clocks are wrong. Also be sure that you are correctly specifying your video card chipset, as well as other options for the Device section of XF86Config. Be absolutely certain that you are using the right X server and that /usr/X11R6/bin/X is a symbolic link to this server.
If all else fails, try to start X ``bare''; that is, use a command such as:
X > /tmp/x.out 2>&1
You can then kill the X server (using the -- key combination) and examine the contents of /tmp/x.out. The X server will report any warnings or errors---for example, if your video card doesn't have a dot clock corresponding to a mode supported by your monitor.
The file VideoModes.doc included in the XFree86 distribution contains many hints for tweaking the values in your XF86Config file.
Remember that you can use -- and -- to switch between the video modes listed on the Modes line of the Screen section of XF86Config. If the highest resolution mode doesn't look right, try switching to lower resolutions. This will let you know, at least, that those parts of your X configuration are working correctly.
Also, check the vertical and horizontal size/hold knobs on your monitor. In many cases it is necessary to adjust these when starting up X. For example, if the display seems to be shifted slightly to one side, you can usually correct this using the monitor controls.
The USENET newsgroup comp.windows.x.i386unix is devoted to discussions about XFree86. It might be a good idea to watch that newsgroup for postings relating to your video configuration---you might run across someone with the same problems as your own.