I could find no obvious reason why my Windows 7 firewall was blocking internet access from my Windows Device Emulator to a server application running on the host PC. So, while the Windows CE application was running on the device emulator, trying to get connected with the server application on the host PC, I turned off the firewall, and as I expected, the Windows CE application could all of the sudden connect to the server.
What surprised me a bit, however, was that turning on the firewall again, let the Windows CE application continue to connect/disconnect to the server on the host PC. Now, after each reboot of the host PC, I always start up the device emulator, let the Windows CE application try to connect, start my server, and then turn off the firewall and then on again. It works every time.