When you have servers in the DMZ that are members of your internal AD (not best practice ok.. ) .. you find yourself shooting holes in the firewall to allow RPC, SMB and other protocols. In that case perhaps an IPSEC tunnel can help you out.. when you use a tunnel between your internal and DMZ hosts, the firewall only has to allow UDP 500 and ESP protocol (protocol 50). No high ports required. To set it up use the following guide.

First of all we need to create IPSEC policies on the domain controller(s) and on the DMZ server. In our example we have a MOSS server in the DMZ that needs to be accessible from the internet. The MOSS server is a member of the domain and the domain controller is on the internal network behind an ISA firewall. Off course the MOSS server itself again is protected by a firewall from the internet.

We need to create a policy on the domain controller that tells the domain controller all traffic towards the MOSS server needs to be inside an IPSEC tunnel. All other traffic needs to remain unaffected. In this case we use the local policies, but you can also use central group policies. Note that the MOSS server already needs to be member of the domain when we place it in the DMZ. Go to start-run and type MMC. Use CTRL-M to add new snap-ins. Select the IP Security Policies on the local computer. You will see three standard policies: Server (Request Security), Client (Respond only) and Secure Server (require Security). We will create a new policy called Secure Specific traffic. So right click (or use Action on the menu) and select Create IPSecurity policy and the wizard will pop-up. Click Next, type the name and click next. De-select activate the default response rule (we will activate it in a later stage) and click next. Leave the Edit properties selected and click next.

The list with IP Security rules will be empty apart from the <Dynamic> rule. Click add to add new rules to our policy. Now the policy is defined by two properties. The traffic (source-target-port) and the action that is linked to the traffic (allow, deny, secure). First we need to make sure the domain controller remains accessible for all systems in the internal network. We therefore apply the first rule, allow any traffic from any source. Click next on the welcome screen. Now on the Tunnel Endpoint select This rule does not specify a tunnel, since we are not creating a VPN tunnel.

Then select All network connections or LAN to specify the network type. Select the predefined All IP traffic and click next. For the action click Permit and click next. Deselect edit properties on the next page and click next. You will now see the new IP Filter List in the overview.

Click Add to add the new rule for connections to the MOSS server. (remember we need to tunnel this traffic). Click Add, click next, again select This rule does not specify a tunnel and click next click all network connections and then on the IP Filter list click add. Give the name “Tunnel this traffic” and click add to specify source target and protocol/port.

On the IP Filter wizard click next, give a description and click next. For the source address select My IP Address, and click next. For the destination select A specific IP Address and enter the ip address of the MOSS server. Select Any protocol and note that the port remains unselect able. Click next, and click finish. Note that your traffic is now in the IP Filter list window. Click Ok. Now Select the Tunnel this traffic Filter list and click next. You will then be showed the default actions. Click add to add our own new action (tunnel only). Click next and then enter a name for the action. Type Tunnel and click next. Select Negotiate security in the General options window. What to do when computers do not support IPSec, select Do not communicate with computers that do not support IPSec. For the IP Traffic security select Integrity only. And click next and then finish. Now select the Tunnel action and click next. Then for the authentication method use Kerberos V5 protocol or use a certificate or a preshared key (when the DMZ server is not a member of the domain). Click next and then click finish.

This is only one part, you must do the same (and specify the domain controller as destination address). You can also include multiple rules so that only specific traffic is allowed (HTTPS inbound for example) for internet access etc. If you have an ISA firewall open IPSEC ESP and IKE client and DNS protocols (do not forget to set the ISA in routed mode). Now the last one (DNS) is a bit strange isn’t it.. we said we would tunnel ALL the traffic, not all except DNS. But when a system startsup, the IPSEC driver is not directly initialized, and normal traffic will occur. You can change this with a registry key if you like, see technet: http://technet.microsoft.com/en-us/library/cc783039.aspx

One thing to keep in mind is that it can take some time for IPSec to be initialized and that traffic is flowing. If you still have trouble with the setup, you can also use the Preshared key option. That actually looks more stable than the Kerberos V5 option…