You must have 3.7 and your actual database installed somewhere. I would leave it there. I would then back it up multiple times both using AC's backup and a backup program. Make sure you use the AC utility to clean up and compact the database.
Install 4.0 to your new server. You shouldn't need to use the database as the database will be blank. During the instll, it will ask you to migrate the data. Browse to the data and migrate it. It will then ask for the tuner. IF YOU GET ANY ERROR MESSAGES AT ANY POINT, POST THEM HERE.
If you are able to migrate your data over and run the database tuner so that everything is working, then be happy, jump in the air, go get a beer and leave it along for a day.
Next day, run AC Utility again. Make a backup and put it on your desktop. Install v5 preferably to a different drive. After install, go through setup with your same name and demographics. Some users like to even make a fake patient. Do a note with the fake patient to show that it works. If it does, then simply restore the database using the backup you created from v4.
I know this is an extra step, but in going from 3.7 to 5, you are moving from Access to SQL Server, and it may be easier to do it in two steps.
Otherwise, install v5 on the server, it will ask for you to migrate your database from 3.7. But, you have tried that with issues. If you go that route and there are issues, post the EXACT error message(s) here.
I am up at 1 am generally, so I could try to remote in and help, although it has been awhile since I upgraded.
My other HUGE suggestion would be to go to:
http://www.trainsignal.com/Small-Business-Server-2008-Training.aspxand watch this video over and over and over. Setting up a domain is easy, but it is not easy. Even
http://connect can be a little difficult at times. You will now be using Active Directory.
Do you have a static IP. To get the full use of a domain, you may want a static IP.
