Errors with Custom vCenter Appliance 6.7 Certificates and SRM 8.1 (Error 25239)

In this post I decided to talk about two topics regarding vCenter certificates. VMware has improved a lot, but there are still a few bugs and some strange errors. I will show you how to import new custom certificates into vCenter 6.7 and how to resolve an error 25239 installing SRM 8.1. I had this error in a customer and the troubleshoot was quite difficult and the problem was certificate related. So here are my notes about the two topics:

• Installing custom certificates into vCenter 6.7 and troubleshooting
• Installing SRM 8.1 with custom certificates in vCenter 6.7 and troubleshooting

Installing custom certificates into vCenter 6.7

The main objective is to have a HTTPS connection with no errors like this:

  First of all, you will need three certificates in the following format:

• root.cer of the Root CA (it can be internal or external CA). If you have an intermediate certificate, you must join the two certificates. Please check this article to do so;
• certificate.cer generated by the Root/Intermediate CA. This is the certificate with the DNS name(s) that you want to change in vCenter. Something like vcenter.domain.local;
• certificate.key of the main certificate (this is the private key)

If you don't have this format extension, you can always use OpenSSL to convert them: Download

Here are some examples:

• Convert PFX to CRT/CER with certificate and private key

openssl pkcs12 -in certificate.pfx -clcerts -nokeys -out certificate.cer
openssl pkcs12 -in certificate.pfx -nocerts -out certificate.key

• Convert PEM to PFX

openssl pkcs12 -export -out certificate.pfx -inkey certificate.key -in certificate.crt -certfile CACert.crt

• Convert to multiple formats using a WebTool
  

With this certificates you are ready to start:

     1. SSH to the vCenter server
     2. Run the following command to enable and access the Bash shell:

shell.set --enable True
shell 

     3. In the Bash shell, run the following command to change the default shell to Bash:

chsh -s "/bin/bash" root

     4. Copy the three certificates to vCenter server with WinSCP or similar. Copy them to /root/cert/
     5. Return to the Appliance Shell by running the following command:

chsh -s /bin/appliancesh root 

     6. Enter the following command to manage the certificates

/usr/lib/vmware-vmca/bin/certificate-manager 

     7. Choose option 1
     8. Provide the username and password of vCenter administrator
     9. Choose option 2
   10. Enter the full path to the main certificate.cer
   11. Enter the full path to the main certificate.key
   12. Enter the full path to the root CA root.cer

It can take a while, because all the services will be stopped and started, but in the end you will have your vCenter with HTTPS connection with the DNS name(s) you choose.

I had one error with my certificate:

Command Output: /root/cert/certificate.cer: OK
Status : 10% Completed [Replacing Machine SSL Cert...]
Previous MACHINE_SSL_CERT Subject Alternative Name does not match new MACHINE_SSL_CERTIFICATE Subject Alternative Name
Performing rollback of Machine SSL Cert...
Get site nameus : 0% Completed [Rollback Machine SSL Cert...]

For me that was not obvious, because my SAN was correct. So I investigated further and found that the new certificate had this Subject Alternative Names:

• hostname.domain.com
• cname.domain.com

But vCenter was expecting the following SANs:

• hostname(s)
• ipv6address

So, the simple solution is to create another certificate with IPv6 SANs (not for me, because it was a external certificate) or just disable the SAN check in the Python Script!

     1. Edit with vi the file: /usr/lib/vmware-vmca/bin/certificate-manager
     2. Comment line 594 and line 595 (see below)

#        if var.strip() in ['1']:
#            iscomparerequired = compare_certificate_san
or
#        if var.strip() in ['1']:
#            iscomparerequired = compare_certificate_san_to_pnid(cert_file)

    3. Re-run the certificate manager
    4. Check if everything is ok.

Installing SRM 8.1 with custom certificates in vCenter 6.7

I have installed many times SRM and this is a straightforward process, but this time I had serious problems installing SRM 8.1 with the vCenter certificates.

The installation did not finish and the error was this one:

Error 25239: Failed to configure Site Recovery Manager. Failed to acquire token from SSO Server at https://SERVER_VC/sts/STSService/vsphere.local

It was very strange because the certificate in vCenter was OK and all the services were running fine. I tried several things:

• Reinstall SRM Server and install all Windows Updates;
• Import root and intermediate certificates of vCenter to the local machine;
• Install SRM with domain account
• etc.

No luck... the same error all the times. The Log keep saying "Failed to acquire token from SSO Server...". I started to look deep at vCenter server.

I will not go in detail about the troubleshoot I took, but I stumbled in a VMware article that says the following:

Some solutions, such as VMware vCenter Site Recovery Manager, VMware vSphere Replication, or VMware vCenter Support Assistant are always installed on a different machine than the vCenter Server system.
If you replace the machine SSL certificate of a vCenter Server system with an embedded Platform Services Controller, a connection error results when the solution attempts to connect to the vCenter Server system. The reason is that the vCenter Server system uses a new certificate, but the corresponding registration with the VMware Lookup Service is not updated. When solutions connect to vCenter Server, they use the service registration information, which includes the service URL and the sslTrust string. The sslTrust string is the Base 64 encoded certificate. By default, the old certificate remains part of the service registration even if you successfully replace the vCenter Server certificate

It must be it!
https://kb.vmware.com/s/article/2109074
(check the procedure details)

To resolve this issue when using the Platform Services Controller UI to replace the certificates, run the ls_update_certs.py script on the Platform Services Controller. When you run the script, you pass in the old certificate and the new certificate.

Notes:

• Run this script always on the Platform Services Controller.
• To run the script, you need the thumbprint of the old vCenter Server certificate and you need the new certificate. You must upload these files to the Platform Services Controller before you run the script.
• Ensure to back up your existing certificates before you run the script.
• Run this script each time you replace a certificate.

The main steps are the following (with vCenter Appliance):

     1. Get the current sslTrust anchor stored for the Platform Services Controller:

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

     2. Run this command to get the current SSL certificate used on port 443 on the Platform Services Controller:

echo | openssl s_client -connect localhost:443

     3. Using the output from the openssl s_client and the lstool.py, verify if the returned SSL certificates match for your vCenter Server with embedded Platform Services Controller. If they do match, you do not need to continue. If they do not match, please proceed

     4. Retrieving the Old Certificate

You can retrieve the old certificate using the Managed Object Browser (MOB) or from the backup store. Backup store contents changes after each certificate replacement operation; using the MOB is a more reliable option.
To open the MOB, go to https://vc_with_embedded_psc.example.com/lookupservice/mob?moid=ServiceRegistration&method=List in a browser.
Log in with the administrator@vsphere.local username and password when prompted.

In the filterCriteria text field, leave only the tags <filterCriteria></filterCriteria> and click Invoke Method. The ArrayOfLookupServiceRegistrationInfo object is displayed.
Search (Ctlr+F) for vc1.example.com on the page.
Find the value of the corresponding sslTrust field. The content of that field is the Base64 encoded string of the old certificate. Any of the occurrences of vc1.example.com and Base64 encoded strings is acceptable.

Copy the Base64 encoded string to a file and save the file as old_machine.txt.
Open old_machine.txt in a text editor.
Append -----BEGIN CERTIFICATE----- to the beginning of the text string, and append -----END CERTIFICATE----- to the end of the text string. Add a carriage return after the 64th character of each line of the contents copied from the sslTrust field.

For Example:

-----BEGIN CERTIFICATE-----
LIIDeDCCAmCgAwIBAgIJAP7kGwWSSd0yMA0GCSqGSIb3DQEBCwUAMGgxCzAJBgNV
PAMMAkNBMRcwFQYKCZImiZPyLGQBGRYHdnNwaGVyZTEVMBMGCgmSJomT8ixkARkW
....
PEJ3vlvSRy7l2lvU19upt4O/BAk3ZJ+X5uFtv/4GMdbEVZBCmNDS7Y85NorISiQf
AVy/R2wjP4rNWDfN9DMCcwfPvw/0nFwrpr+0Cg==
-----END CERTIFICATE-----

Save old_machine.txt as old_machine.crt.

     5. Run this command to get the thumbprint:

openssl x509 -in /certificates/old_machine.crt -noout -shal -fingerprint

The Thumbprint is like this:
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX

     6. Retrieving the New Certificate

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT

Export the certificate to a file with this command: 

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /certificates/new_machine.crt

Move or upload the certificate to the Platform Services controller via WinSCP or another SCP client.

     7. Update all services with the New Certificate Running the ls_update_certs.py Script

Run the ls_update_certs.py script on the Platform Services Controller after replacing the vCenter Server certificate. To successfully run the script, you must have both the thumbprint of the old vCenter Server certificate and the new vCenter Server certificate.

The ls_update_certs script is located at /usr/lib/vmidentity/tools/scripts/ls_update_certs.py

Run this command:

python ls_update_certs.py --url Lookup_Service_FQDN_Platform_Services_Controller --fingerprint Old_Certificate_Fingerprint_Task_5 --certfile New_Certificate_Task_6 --user Administrator@vsphere.local --password XXXXX

For Example (do not copy the thumbprint and the FQDN):

python ls_update_certs.py --url https://fqdn_vc_server/lookupservice/sdk --fingerprint XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX --certfile /root/cert/new.crt --user administrator@vsphere.local --password XXXXX

     8. Run steps 1 and 2 to verify if the returned SSL certificates match

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

echo | openssl s_client -connect localhost:443

If they match, you are done! Please try to install SRM 8.1 again and this time you will have no errors regarding vCenter certificates.

Rule of Thumb. When you change vCenter certificates, validate that all services (mainly the Lookup Service) are with the new certificate with the commands I showed you before.

See you Next Year!