Exchange Server 2007 Lost - How to Recover Installation and Definitions on a New Server

To end the year smoothly (or not!), I recently had a costumer that lost the Physical Exchange server because a corruption of the disks after a power failure.
The worst fear of any Microsoft Exchange Server administrator is a complete crash of the Exchange server.
First thing to ask.. Backups! Ok, we have backups from the Exchange databases, but not from the Operational System and Exchange Installation.
In this post I'll show how to recover step by step a exchange server (in this case Exchange 2007 with Windows Server 2008. It applies to Exchange 2010 and Windows Server 2003 too) with the following roles: Mailbox, Client Access, Hub Transport, and Management Tools from scratch.

So, let’s apply that how-to treatment to the most difficult scenario of an Exchange recovery, which is the inability to replace the crashed Exchange system on the same hardware. If the hardware is still operational you can use the same server, if not, you have to rebuild your server on new hardware. This article applies to both cases.
Fortunately, most of the configuration details about your dead Exchange server are stored in Active Directory (AD), and for the purposes of this article, I'll assume that your AD infrastructure is still intact. (If that's not the case, you must recover your AD infrastructure before rebuilding your Exchange server).

• If you are using the same or a new server, or you only replace the SmartArray, you must create the RAID volumes first. Make sure that you have an adequate number of drives to duplicate the exact same drive letters, with storage space on each drive letter equal to or greater than that of the original server. So you can restore the exchange databases later. If you don’t know where Exchange was installed on the old server, you can use ADSIEdit.msc (installed with the Windows Support Tools or here) on any server in AD.
  • Expand Configuration, CN=Configuration, DC=<domain>, DC=<domain_ext>, CN=Services, CN=Microsoft Exchange, CN=<company_name>, CN=Administrative Groups, CN=Exchange Administrative Group (admin_group_id), CN=Servers,CN=<Exchange_Server>;
  • Right-click the server name, and select Properties. Select the Show only attributes that have values check box and click on the Value column to sort by value. Scroll down until you see the full path of the Exchange server attributes, and make a note of each drive letter and path. You can also view the path of each storage group and database on the server by expanding CN=<server_name>, CN=Information Store, CN=<storage_group> and examining the properties of each storage group and database stored on the server.

• Install the same OS as the failed server with all the patches (This is important!). Exchange 2007 will run on Windows Server 2008 x64 or Windows Server 2003 x64. Install the latest service pack with all the patches. Avoid the temptation to upgrade the OS during the disaster-recovery process. The last thing you want to troubleshoot are new OS problems that might arise because you decided to upgrade the OS during the recovery process. Installing the same OS reduces the chance of having problems with backup, antivirus, or other related programs during the recovery process. If you don't know or remember what version did you had, you can go to Active Directory Users and Computers and see the version on the Computer Object.

• Name the new server the same as the failed server with the same IP address configuration. This step is very important because when you perform an Exchange re-installation, the system will use the server name to gather the failed server’s configuration from AD. If you can’t remember the failed server’s name or IP address, you can look that up on any DC that’s running AD-Integrated DNS. Try to ping the name of the Exchange server to know its IP Address or a reverse lookup to know its hostname;

• Reset the computer account in Active Directory. Before you join the domain, start the Microsoft Management Console (MMC) AD Users and Computers snap-in, locate the Exchange system, right-click it, and select Reset. Doing so will let you use the old server’s name to join the new server to the domain;

• Join the domain and restart the server;

• Stop inbound mail flow into the new server before you start the re-installation of Exchange. You can typically accomplish this at the firewall or spam-filtering appliance. Stopping mail flow will prevent you from losing any Internet inbound messages that could be overwritten during the recovery process;

• Install the necessary prerequisites and roles on the server for Exchange. If you’re running Exchange 2007 on Windows Server 2003 (if not, move to the next step), these include the following:
  • MMC 3.0;
  • NET 2.0 with SP1;
  • Windows PowerShell;
  • IIS, World Wide Web, and Common Files;
  • If you're running the Unified Messaging Role, you'll also need: Media Encoder 9 x64, Codex Patch KB917312 and Core XML 6 Servers.

• If you’re running Exchange 2007 on Windows 2008 with the Mailbox, Client Access, Hub Transport, and Management Tools, you can script these requirements through the following commands:
  • ServerManagerCmd -i PowerShell
  • ServerManagerCmd -i Web-Server
  • ServerManagerCmd -i Web-ISAPI-Ext
  • ServerManagerCmd -i Web-Metabase
  • ServerManagerCmd -i Web-Lgcy-Mgmt-Console
  • ServerManagerCmd -i Web-Basic-Auth
  • ServerManagerCmd -i Web-Digest-Auth
  • ServerManagerCmd -i Web-Windows-Auth
  • ServerManagerCmd -i Web-Dyn-Compression
  • ServerManagerCmd -i RPC-over-HTTP-proxy

• Run the Exchange 2007 installation with "Setup /m:RecoverServer". Install Exchange 2007 with this switch is similar to performing a disaster-recovery installation on Exchange 2003. Most of the Exchange settings are stored in AD. The /m:RecoverServer switch tells the Exchange Setup program to query AD for the Exchange settings based on server name. Setup will run through the prerequisite check, and if everything is in place, Exchange 2007 should be installed on your recovered server. After a successful installation, reboot the server and at this point, you should have your Exchange implementation restored without any current data. Open the Exchange Management Console (EMC), review everything, and verify that the server was properly installed and with all the configurations on the Hub Transport and Client Access. By default, the empty Exchange databases dismounted after the Exchange installation.

• For any databases that you plan to restore, select the properties of the database and ensure that the "This database can be overwritten by a restore" check box is selected.

• Install the backup agent with the same configuration you had on the original server;

• Reinstall the SSL certificates. If you had a commercial SSL certificate on the failed server, you must reinstall it. If you didn't export the certificate to a PFX file, have your SSL vendor reissue the certificate or obtain a new SSL certificate. If you have to order a new certificate, make sure that you obtain an SSL certificate that's capable of multiple identities. Typically, you'll need the following identities for the SSL certificate: internal Fully Qualified Domain Name (FQDN) of the Exchange server, external FQDN of the Exchange server, and Autodiscover.<domain_name>.com. If you only had an internal certificate, you must create a new one on the internal CA (not described in this article);

• Install VSS Snapshot Patch KB940349. If you don't install this patch, the Exchange database-restore job might fail with the error Final error: 0xe00fed5, A Failure occurred initializing for restore;

Now you have two options, Restore from a previous backup or, if didn't lose the exchange databases (only operating system)  you can reuse them:

• Restore the databases to the Exchange server using your backup software. Ensure that the existing databases are dismounted otherwise, the restore will fail. After the restore is complete, verify that the database is mounted and that the mailboxes and public folders were properly restored. (Using the EMC, verify that the "Do not mount this database at startup" check box is cleared;
• Copy the Databases to the original location as you saw in the Active Directory on the previous topic and try to mount the databases;
• If you have errors like "-1216 (JET_errAttachedDatabaseMismatch)" or "hr=0x80004005, ec=-1216" and "hr=0x80004005, ec=-455" continue on this topic, otherwise go to the next point;
• Perform the eseutil /mh "Path of the database" as indicated below and check the state of the database;
• If the state is in clean shutdown, move all the log files from the Transaction logs folder location and then mount the store:

• If the state is in Dirty shutdown as mentioned below, check if the log files that is indicated as Logs required is available

• To make sure that the log files that is required is in a Clean state, you can perform eseutil /ml "Path of the log files\log prefix" . This command will help you check the health of all the log files in the location;
• If the log files are healthy, then perform the Soft recovery with the command eseutil /r <Log Prefix> /l "Path of the log files" /d "Path of the database
• Once the command completes successfully, mount the stores.
• If you still get errors you must try a Hard Reset: eseutil /p "Path to EDB Files\DATABASE.edb" 
• When you are prompted to confirm this operation, choose OK;
• It can take a while to complete. DO not cancel this task or you can corrupt the databases.;
• Repair process completed;
• Mount the Databases.

• The certificate that we earlier import to the server, must be used with the Exchange Services:
  • Get-ExchangeCertificate | fl
  • Note the Thumbprint of our external/internal certificate that has our DNS names like autodiscover, webmail, etc.
  • Enable-ExchangeCertificate -Thumbprint <put_here_the_certificate_thumbprint> -Services "SMTP, IMAP, POP, IIS"

• Now its time to enter the right External and Internal names of the different services that Exchange provides:
  • Set-ClientAccessServer –Identity <CAS Server Name> -AutoDiscoverServiceInternalUri: <Internal URL>
  • Get-OABVirtualDirectory and Set-OABVirtualDirectory to list and set up the OAB URLs
  • Set-WebServicesVirtualDirectory –Identity “<EWS Name>” –InternalUrl: https://url.domain.local/EWS/Exchange.asmx
  • Set-UMVirtualDirectory –Identity: “<UM Virtual Directory>” –InternalURL: <URL/UnfiiedMessaging/Service.asmx>

• If the Outlook Anywhere is not configured , you must change the external name that matches your externa certificate and change the authentication to Basic;

• If there where redirects from HTTP to HTTPS or/and HTTP://webmail.domain.com/ to HTTPS://webmail.domain.com/owa you must configure that on the IIS;

• Reestablish mail flow from the Internet;

• Test the restored mail server to verify that it's properly functioning. This process might include (but is not limited to) verifying Autodiscover functionality, ensuring that all mailbox and public folder information was properly restored, checking mail flow to and from internal Exchange users (as well as to and from the Internet), verifying ActiveSync functionality, checking Outlook Web Access (OWA) functionality, ensuring Unified Messaging functionality (if applicable), and verifying relay functionality for servers that use the Exchange server like SharePoint servers and other servers that use this server to send mail.