Upgrade Veeam Backup & Replication to v12 and SQL Server to PostgreSQL

In this Post I will show you how to upgrade Veeam Backup and Replication from version 11a to v12, including database migration from SQL Server to PostgreSQL (the default database in this new Veeam release).

I used RTM version released in 30 january 2023 (I think there won't be big differences for the GA version):

• Previous Versions: 11.0.1.1261
• Upgrade Version: 12.0.0.1402

The Upgrade and Migration are done in three steps:

1. Upgrade v11a to v12 including all components (console, explorers, agents and plugins);
2. Database Backup and Restore from SQL Server to PostgreSQL;
3. Cleanup SQL Server databases and files.
   

Since it will be necessary to stop services and Backup Jobs, it is convenient that this process be carried out in a period where there is no backup window. This whole process takes about 2 hours (depending on the infrastructure).

Note: I will not cover the Veeam Backup Enterprise Manager Upgrade.

Let's start!

Hardening and Secure Veeam Backup and Replication Servers

In this post I will show you how to secure/hardening your Veeam Servers in addition to following the 3-2-1-1-0 security standard.

3: Maintain at least 3 copies of your data

By three copies, I mean that in addition to your primary data, you should also have at least two more backups of the same data.

2: Store the backups on 2 different media

Store one of the copies on internal hard disk drives and the other copy on removable storage media (tapes, external hard disk drives, cloud-storage, etc.)

1: Store at least 1 of the copies at an offsite location

One copy of the backups away from the physical location where the primary data and primary backup is located. If the company has no remote or branch office, an alternative could be saving a copy at a service provider in a private cloud or saving a copy in the public cloud.

1: Store at least 1 of the copies offline or immutable

Keep at least one copy of the backups offline (USB Drives, tapes ejected) or immutable (Linux with Veeam immutability or object storage with immutability).

0: Be sure to have verified backups without errors

Backups are only as good as they are being verified.

Hardening is about securing the infrastructure against attacks, by reducing its attack surface and thus eliminating as many risks as possible. One of the main measures in hardening is removing all non-essential software programs and utilities from the deployed Veeam components. While these components may offer useful features to the administrator, if they provide “back-door” access to the system, they must be removed during the hardening process.
Protecting your infrastructure successfully is all about understanding the current attack vectors; what and whom you are protecting, your Veeam infrastructure, against. If you know what and whom you are protecting against, makes it easier to take the correct countermeasures. One of those countermeasures is hardening.

Looking at the different Veeam Backup & Replication components you must protect the following components:

• Veeam Backup server
  
• User Accounts
• Backup repositories
• Backup data flows
  

Consider the Veeam Backup & Replication server to be the Number 1 target on your infrastructure and it should have very restricted access. As a general rule, the backup server is the single greatest target a hacker can claim on your network/infrastructure. Also the backup repositories which holds the backup files are a primary target.

There are two options to secure all the Veeam Servers. Let's see the details.

NetApp - Convert 7-Mode to Cluster-Mode and Upgrade ONTAP 8.X to 9.8

 

In this first post of 2022 I will show you how to convert from Data ONTAP 7-Mode to Data ONTAP Cluster Mode 8.X (and later versions). In this step-by-step, the source version is v8.2 and the final version will be v9.8. Be aware that this process will wipe all the data. So, you first need to backup all the data or move it to another repository.

First you need to check if yout Netapp has one or two controllers.

• One Controller: You don't need to perform any additional step;
  
• Two Controllers: You need to connect both controllers with a DAC Cable.
  

There are several in-between version upgrades that we need to do first, so we can go to v9.8. Let´s first see the Upgrade compatibility.

VMware Metro Cluster - DRS Automation with TAGS

 

Hello guys, in this post I will show you a how to use TAGs in VMware to distinguish the Site where the virtual machines are running. In this way,  and using PowerCLI Scripts we are able to allocate the VM to the Host of the respective site with the help of DRS (Distributed Resource Scheduler).
It is possible to make these configurations without using TAGs, but I think that this way it is more flexible, simple and elegant. The only requirement is to have Metro Cluster working properly (it's not in this scope how to configure a vSphere Metro Cluster) and the vSphere Enterprise Plus license on every Host.

This is very useful if you have a Metro Cluster, and you want that VMs that are on Site 1 Storage to run on the Hosts in Site 1 and vice-versa for Site 2. If you don't use this feature, you may end up having VMs on Site 1 Storage running on Site 2 Hosts and vice-versa.

Unfortunately, it's not yet possible to make these settings automatically in the vCenter. (I hope that VMware in future releases will place the option of TAGs directly in the DRS, instead of the name of the VMs). So let's deep dive to the PowerCLI Scripts.

HPE 3PAR - Remove a Cage Completely or Some Disks Without Disruption

 

In this Post I will show you how to remove one or more Cages completely or some disks from a 3PAR Storage system. This process involve some risks but it can be done completely Online and without disruption.

Let's first see the requirements for this operation.

Simplifying ADConnect Auth/Sync - On-Premises <-> Azure AD

In this post I will show you what is Azure AD Connect (AADC) and all the options to synchronize your on-premises environment to Azure Active Directory (AAD). This is very relevant because you need this configurations for Microsoft 365 (Office 365), so you can have all of your users, groups and/or devices synchronized in both directions. This is mandatory when we want to have objects with the same SID and same credentials in both environments. For instance, if one user reset or change his password in Azure AD, this new password is automatically synchronized to On-Premises Active Directory (Write-Back feature).

Azure AD Connect is required for all authentication/sync methods we will cover today:

• Password Hash Synchronization
• Pass-through Authentication (PTA)
• Active Directory Federation Services (2019)
• Additional Feature - Seamless Single Sign-On

Let's see the requirements and then all the authentication/sync methods functionalities in depth.

VMware Horizon with HAProxy Cluster LoadBalancer

It's been a while and since I had this post in draft for a long time I finally decided to finish it and publish it. What I'm going to show you is how to make a HAProxy Cluster for VMware Horizon connections with Load Balancing (LB) for two or more active VMware Horizon Connection Servers.

This LB Cluster solution is free (Linux environments) and can be used for other solutions that do not use the same ports (in this case 80, 443 and 8443). In turn, Connection Servers communicate with vCenter to redirect users to the correct VDI Pools after Active Directory authentication.
Let's see how to configure HAProxy redundantly using two CentOS 7 servers.

Format, Extend/Reduce and Create Volume Groups / Logical Volumes in Linux

Recently I had to set up some linux volumes and partitions in a production environment. I have compiled some points that I find relevant.

In this post I will show you how to do the following configurations in Linux Operating Systems:

• Add a new Physical Volume and Create Partition;
• Create new Volume Group and Logical Volume with one or more disks;
• Extend Volume Group with new Logical Volume;
• Extend Volume Group and the Logical Volume;
• Reduce a Logical Volume;
• Mount/Umount NFS Volumes

3PAR Procedures - Replace Disk, Remove Disk & Change Disk Position - Rebalance

In the first Post of the year I will show you three procedures regarding disk failure, remove a disk from the CPG and rebalance disk position in a 3PAR array.

• Replace a failed disk in 3PAR
• Objective: Remove failed disk and replace with a new one
• Remove disks from the CPG and rebalance the data
• Objective: Remove a disk from the 3PAR. The disk will be in New state and you can use it in another 3PAR
• Change disk position in 3PAR so you can rebalance data in the array
• Objective: Maximize performance of your array when you upgrade the 3PAR with new cages or disks

Let's see how to do this three procedures.

Errors with Custom vCenter Appliance 6.7 Certificates and SRM 8.1 (Error 25239)

In this post I decided to talk about two topics regarding vCenter certificates. VMware has improved a lot, but there are still a few bugs and some strange errors. I will show you how to import new custom certificates into vCenter 6.7 and how to resolve an error 25239 installing SRM 8.1. I had this error in a customer and the troubleshoot was quite difficult and the problem was certificate related. So here are my notes about the two topics:

• Installing custom certificates into vCenter 6.7 and troubleshooting
• Installing SRM 8.1 with custom certificates in vCenter 6.7 and troubleshooting

Import Custom 3PAR SSMC Certificates

This is the web based version of the classic 3PAR Management Console application (if you don't use it already, you might know that the old versiont is discontinued).
Today I will show you how to change the default certificates from 3PAR SSMC in a windows platform. You can use external or internal certificates from your Certificate Authority.

All the tools you need to complete this task are pretty well already on the server where you installed the SSMC software.

I think from another post I read that when you upgrade the SSMC version this will overwrite the certificate and you will have to do this again, i have not tested this yet.

Let's see how to do it!

VMware Troubleshoot Commands, Logs and Performance - ESXTOP

Today in this post I will show you some commands and logs to try to solve some errors in a VMware environment. I'll give special attention to a very useful command called esxtop

 

What I'm going to show is the following: 

• Important Log Files

• Useful ESXi Commands

• Some ESXi Configuration Files

• ESXTOP Command

Let's Begin!

Upgrade Firmware Brocade SAN Switch

 

In this post I will show you how to Upgrade the Brocade Switches Firmware.

Requirements:

• Filezilla FTP Server (or similar)
• Filezilla FTP Client (or similar)
• Putty
• Java JRE (the version will depend of the Switch Brocade. Probably you will need version 7)
• Log on credential for Brocade website or vendor website
• Downloaded upgrade firmware. Read the Release Notes because you might need more than one firmware

Let's start with the Upgrade Paths and then with the procedures.

Step-by-Step - Cisco UCS Firmware Upgrade

 

I once had to upgrade a Cisco UCS Manager and the process went very smoothly, without any downtime and I documented all the procedures.

For those who do not know this system here is a brief description:

Cisco UCS Manager supports the entire Cisco UCS server and Cisco HyperFlex Series hyperconverged infrastructure portfolios. It enables server, fabric, and storage provisioning as well as, device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection. You can extend the benefits of Cisco UCS Manager globally across an enterprise to thousands of servers in multiple domains with Cisco UCS Central Software.

More information:

https://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

 

In this post I will show you how to Upgrade UCS 2.2(3c) to version 2.2(8f), but it also applies to other versions. I'm  going to show you how to Upgrade the infrastructure (UCS) software, Fabric Interconnects and I/O modules firmware in a two-stage operation.

So, let's start! 

Domain Controllers - DCDiag & Repadmin

All system engineers have experienced replication problems once in a lifetime.Today I will show you some basic commands to Troubleshoot and Diagnose Domain Controllers and replications. I'm talking about DCDiag and Repadmin.

DCDiag:

This command-line tool analyzes the status of one or all domain controllers in a forest and reports all problems to provide assistance in troubleshooting. DCDiag.exe consists of a variety of tests that can run either individually or as part of a set to check the state of the domain controller. Here are some examples:

1. Preparing to install or migrate to Exchange;
2. Checking FSMO roles;
3. Troubleshooting Group Policies;
4. Investigating Active Directory not replicating frssysvol error;
5. Running down Kerberos authentication problems;
6. Resetting the Directory Service Administrator's password;
7. Fixing a servers Service Principle Name (SPN) error.

Repadmin

Performs replication-related tasks, such as managing and modifying replication topology, forcing replication events, and viewing replication metadata and updated vectors.

Let's see some examples and a Script to automize all Domain Controllers replication.

SCCM - Failed to validate content hash on Distribution Point. Yellow Warning

In SCCM when there are many Distribution Points with different published Software, it may happen that there are validation failures and that WMI repository is inconsistent. The events are represented as in the following table:

 

SCCM by default does not have any tools that allow you to resynchronize the entire WMI repository without removing the Distribution Point Role and adding it again. Let's see how to get around this.

Office 365 Users Not able to view Free/Busy and Room information of On-Premises

The other day I had to validate and solve a problem with calendar synchronizations between Office 365 and Exchange 2010 (and vice verse). I was aware that On-Premises users could see the status of all users calendars, but Office 365 users only could see the Office 365 users status. (Office 365 users were not able to view F/B information of On- Premises user mailboxes).

This scenario is valid for Exchange 2010/2013 and Office 365 hybrid architectures with or without ADFS. The errors that I had were this:

Trying to share a Calender between On-Prem and an Office 365 User:

"One or more users cannot be added to the folder access list"

The outlook/webmail calendar had errors:

"No Information. No free/busy information could be retrieved"

"The recipient's server could not be contacted. Contact your administrator"

Let's see the necessary steps for the resolution.

Create Multiple Mount Points in Multiple Disks in one Datastore VMware

I was in a costumer where I had to create many disks to present to several virtual machines in VMware. The Datastores were few, but I had to create 60 disks in each VM (maximum supported with four SCSI controllers). To increase complexity, all of these Volumes in Windows meant to be Mount Points. To optimize this process I had to create some Scripts to help me.

I splited in different points:

• Script to add multiple disks from one datastore to a VM;
• Script to change all the Disks paths to Round Robin (The Storage was a HPE 3PAR);
• Script to Create all the Windows Mount Points;
• Script to view the Mount Points created.

Let's see how to do it!

Capture Image for SCCM - WDS - MDT using DISM and WinPE

We often want to capture machines with SCCM using the ISO capture and we have very strange errors, namely because of sysprep, Windows 10 applications or even permissions. One of the easiest ways to accomplish this capture is to not use SCCM's ISO and let us capture the image at a low level. To do this, just use the ADK tools already installed on the SCCM server.

In this Post I will show you the steps to accomplish this task!

Upgrade HPE Storage 3PAR to v3.3.1 - Upgrade Service Processor to v5.0

SP5 Release Notes:
https://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-a00002915fr_fr

3PAR 3.3.1 Relase Notes:
https://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-a00017203en_us


Service Processor 5.0 is completely new and there is only one way to upgrade. Lets find out how.