ASite – Primary Site

Server name IP Address Role

ADC 172.168.1.1 Domain Controller

AHC1 172.168.1.2 Hub and CAS

AMBX1 172.168.1.3(MAPI) Mailbox Server

  10.0.1.1(Replication)  

AMBX1 172.168.1.4(MAPI) Mailbox Server

  10.0.1.1(Replication)  

10.0.2.10 -NIC2 NIC1 – 172.168.1.10

Router

 10.0.2.10 -NIC2 NIC1 – 172.168.2.10

BSite – Secondary Site

Server name IP Address Role

BDC 172.168.2.1 Domain Controller

BHC1 172.168.2.2 Hub and CAS

BMBX1 172.168.2.3(MAPI) Mailbox Server

  10.0.2.1(Replication)  

Table 1: LAB Infrastructure

DAG Name DAG01

Witness Server AHC1

Witness Directory C:\DAG01Witness

Alternate Witness Server BHC1

Alternate Directory C:\DAG01Witness

 IPAddress from both the sites 172.168.1.9,172.168.2.9

Table 2: Details for DAG Creation

1. Login to the Mailbox Role Server AMBX1 on a Primary Site. Execute the below
commands on the PowerShell Console to create a new DAG with name DAG01.
2. File Share Witness will be created on the Hub Transport Server (AHC1) and DAG IP
from both the AD sites.
3. Once we have DAG created we also need to configure Alternate file share witness on
the DAG. Alternative File Share witness is configured to point to secondary site.
4. If there is DR and if Secondary sites needs to bring up then this file share witness
directory will be used

New-DatabaseAvailabilityGroup -Name DAG01 -WitnessServer AHC1 -WitnessDirectory

C:\DAG01Witness -DatabaseAvailabilityGroupIPAddresses 172.168.1.9, 172.168.2.9

Figure 1. New DAG Creation

Configure the alternative file share witness using the Powershell cmdlets shown
below

Set-DatabaseAvailabilityGroup -Identity DAG01 -AlternateWitnessDirectory

C:\DAG01Witness -AlternateWitnessServer BHC1
Figure 2. Configuring Alternative File share Witness.

Once we have the DAG created we can able to see the properties of the DAG with the
PowerShell cmdlets.

Figure 3.

get-DatabaseAvailabilityGroup DAG01 | fl

Figure 3. Properties of new DAG Created

ADD members to the DAG

Once we have created DAG we need to ADD members to the DAG. Let’s add all the mailbox
servers from Asite and Bsite into the DAG.
 1. All DAG members must be running the same OS (windows 2008 R2 or Windows 2008
Sp2)
2. When any mailbox Server is added to the DAG, it installs Windows failover clustering
component on to the server and creates a failover cluster and joins the mailbox
server to the newly created cluster.

Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer AMBX1

Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer AMBX2
Add-DatabaseAvailabilityGroupServer -Identity DAG01 -MailboxServer BMBX1
  

Figure 4. Adding AMBX1 to the DAG01

Enabled Database Activation co-ordination

Set-DatabaseAvailabilityGroup -Identity DAG01 -DatacenterActivationMode DagOnly

Configure database copies for the Databases in the DAG. I have created two
Database in each of the mailbox server in Asite (Primary)

Get-MailboxDatabase |?{$_.Server -like "AMBX*"}

Type of Failure
1. Database Failure

2. Server Failure:- Move-ActiveMailboxDatabase -Server AMBX2 -ActivateOnServer

AMBX1

3. Site/Datacenter Failure
 1. The above scenario PAM (Primary Active Manager) helps to bring the database up in
case of the failure.
2. We have defined not to bring the database up in case of failure by setting Activation
bit. So, we have to manually execute some PowerShell cmdlets to bring the services
up and running…

1. how we can fail back once the Primary datacenter come up.
2. Now I have disconnected the Network between the AD sites and brought down all
the Servers in ASites to have complete Datacenter failure.
3. Lets see the status of the DAG01 from the BMBX1 using the below Powershell
cmdlet. Figure 19. Show that the entire Database from Primary site is service down
and Passive copy is in Disconnected state and healthy.
4. We have also defined not to bring Secondary server Database up in case of Primary
Datacenter Failures. This is done using DAC configuration

Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft

-auto

Figure 19. DAG status after the Primary DC Failure

In a our 3 server DAG, cluster quorum is maintained by a node majority – so at this point
with two nodes offline the remaining server cannot hold quorum and this is also a reason
our secondary server database is dismounted and cannot be re-mounted as well.

1. In Figure 20. Marked in red has the details about started mailbox servers and
Stopped Mailbox Servers.
2. Started mailbox servers are the servers which are available for DAG for bringing the
Database online.
3. Stopped mailbox Servers are no longer participating in the DAG.
4. They me be servers which are offline or down because of Datacenter failures.
5. When we are restoring the service on secondary site, ideally all the servers which are
in primary should be marked as stopped and they should not use when the services
are brought online.

Figure 20. DAG details

1. Move the Primary Site Servers into stopped state we need use the below PowerShell
cmdlet. We also have to use the parameter -Configurationonly.
2. You may see some warning and error message because these servers are not
reachable.
Stop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX1
-ConfigurationonlyStop-DatabaseAvailabilityGroup -Identity DAG1 -Mailboxserver AMBX2
-Configurationonly
Figure 21. Stopping the DAG server to stopped Server State

Let’s verify again to see if the server has moved to stopped state. Figure 22. Show AMBX1
and AMBX2 has been moved to stopped state. Now these servers are not available for the
DAG recovery.

Figure 22. DAG status after moving Asite servers into stopped mailbox Servers

1. Next we need to verify and make sure if cluster services are stopped on all the
mailbox servers on Secondary Site.
2. In our Secondary site we have only one mailbox servers BMBX1. So, let’s stop the
cluster service using the command “Net stop Clussvc” or manually stop the cluster
service from services console.

Now we need to restore DAG at the BSite. To restore use the below PowerShell
cmdlet.

Restore-databaseavailablilitygroup cmdlet does the following

1. Custer Quorum will be formed at the new server BMBX1 as old Quorum is no
Longer reachable

3. AMBX1 and AMBX2 nodes will be marked as stopped state and it will evict servers
One by one leaving only one node BMBX1 into the DAG.

Switch to use Alternative file share Witness which was defined while creating the
DAG

Restore-DatabaseAvailabilityGroup -Identity DAG01 -ActiveDirectorySite BSite

Figure 23. Execution result of Restore Database availability group

1. Let’s look at the DAG -status before we continue. Figure 23 should has the new details
like Operational Servers is BMBX1 as we have evicted other servers out.
2. PAM(Primary Active Manager) is operational from BMBX1

Figure 23. Dag PAM and Operation Server Status

Let’s check out how the Failover cluster manager looks. We should see that there is only
one Node BMBX1 and current node hosting cluster is BMBX1 and it’s using the alternative
file share witness BHC1 which is good from Figure 24

Figure 24. Failover Cluster manager Status

Let’s verify again the Mailbox Database copy status using the powershell cmdlet and Figure
25 shows that we have still Databasecopies on BMBX1 is in Disconnected and Healthy state

 
Figure 25. Mailbox Database copy Status

During the DAG configuration we had set activation block on database copies on
BMBX1. To remove the activation block on all the copies, we need to execute the
blow Powershell cmdlets.

Resume-MailboxDatabaseCopy ‘MDB01\BMBX1’Resume-MailboxDatabaseCopy ‘MDB02\BMBX1’Resume-

MailboxDatabaseCopy ‘MDB03\BMBX1’Resume-MailboxDatabaseCopy ‘MDB04\BMBX1’ 
  
Figure 26. Execution result of Resume mailbox Database copy

With this we have resumed all the Mailbox Database copies on BMBX1. Now let’s check the
status. Figure 27 show that DAG status with the entire database mounted on BMBX1 and
serving the email for the users

Figure 27. DAG Status with all the Database copies mounted on BMBX1

If databases are not mounted automatically using above technique then you can use below
Powershell command to manually mount. You bunch of options to troubleshoot and mount
the database.TechNet has more details description on parameters of Move-
Activemailboxdatabase

Move-ActiveMailboxDatabase –Server FQDNofaServerinPrimarySite –ActivateOnServer

FQDNofaServerinDRSite

Fail back to Primary Site

1. Lets bring all the servers at primary site up and as these servers are out of DAG
configuration.
2. It will have no impact on the DAG. Now the DAG has only one server BMBX1.
 3. To fail back to the primary site we need add the Primary Site mailbox servers AMBX1
and AMBX2 into DAG back.
4. To add the server back, below Powershell cmdlet

Start-DatabaseAvailabilitygroup -Identity DAG01 -mailboxServer AMBX1Start-

DatabaseAvailabilitygroup -Identity DAG01 -mailboxServer AMBX2

5.

Figure 28. Execution result of adding AMBX1 and AMBX2 into the DAG

If we see the Database available group status you would find that all the mailbox servers
are in started and operation state.

Figure 30. DAG status after adding mailbox servers back

Execute the below PowerShell cmdlet to set the changes for seed all the changes
from the Active copies and bring the passive copies into the healthy state.

Set-DatabaseAvailabilitygroup -Identity DAG01

Let’s now verify the Mailbox Database copy to make sure that we have all active
copies mounted on BMBX1 and passive copies replicated and also it’s in healthy
state both on AMBX1 and AMBX2.

Get-MailboxDatabaseCopyStatus -Identity MDB0* | select name, status | sort Status | ft –

auto

Figure 31. Mailbox Database copy Status

To bring the respective copies of AMBX1 and AMBX2 up, we need to run

Move-Activemailboxdatabase PowerShell cmdlet and other complete set of cmdlets are

below.

Move-ActiveMailboxDatabase MDB01 -ActivateOnServer AMBX1 -MountDialOverride:

GoodAvailabilityMove-ActiveMailboxDatabase MDB02 -ActivateOnServer AMBX1
-MountDialOverride: GoodAvailabilityMove-ActiveMailboxDatabase MDB03
-ActivateOnServer AMBX2 -MountDialOverride: GoodAvailability
Move-ActiveMailboxDatabase MDB04 -ActivateOnServer AMBX2 -MountDialOverride:
GoodAvailability

Figure 32. Move Active mailbox Database execution result.

Verify and confirm again, to see if we have the entire database moved and mounted on the
primary node and other copies are replicated in Healthy

Figure 33. Mailbox Database copy Status after the recovery of Database at Primary Site

Finally last but not least we wanted to disable automatic activation of database in
secondary (DR) Site. This configuration is very important and it can be again
disabled using the same old below PowerShell cmdlet.

Suspend-MailboxDatabaseCopy -Identity MDB01\BMBX1 -ActivationOnlySuspend-

MailboxDatabaseCopy -Identity MDB02\BMBX1 -ActivationOnlySuspend-
MailboxDatabaseCopy -Identity MDB03\BMBX1 -ActivationOnly
Suspend-MailboxDatabaseCopy -Identity MDB04\BMBX1 -ActivationOnly
Figure 34. Disabling Activation bit on Passive copy of the Database on secondary site

With this we have tried to simulate all different type of failure – Database Failure, Server
failure and Datacenter failure and how to recover back from the failure. DAG has made HA
very easy and quicker to recover. Here we just talked about the DAG and the mailbox
servers and Mailbox Database and the recovery it. Exchange is not just DAG. We have to
plan and design for the failover and fail back of other servers like Hub, Client etc.
This TechNet article has good details on other servers. I hope this article is information and
you can use this in your real life