Configuration Manager 2012 R2 Linux OpenSSL Centrify Error

While installing SCCM Linux clients I have come across issues.  Some are well documented some are not documented at all. It has been one of those fun and continuing adventures in Linux and SCCM.  One of the toughest issues that I discovered had to do with Centrify.  I originally added this information to the Linux SCCM Troubleshooting Wiki.  I felt as though I should try to put this information in as many places as possible since this was one of those "not documented at all" issues.   Also I can add a bit more context to the issue here.

Background:

While installing the Linux SCCM client on several different flavors of Linux I started to experience some issues with some of them.  The client would install but the service would not start.  In the log I was receiving a kernel general protection error.  After scouring the Internet and sending emails back and forth with others, I still did not have an answer as to the cause of the error.  The system logs did not show any issues.  I checked to make sure that the openssl version installed was supported and it was.  But then I decided to look a bit further into the openssl I found something was not right.  Versions of files and packages did not match.

Something seemed a bit off...

AHA! I realized what the issue was...

The problem is that Centrify installs its own version of OpenSSL and then changes some global defaults to the systems OpenSSL. In order for the SCCM client to work it must link its libraries to the Centrify ones and not the system OpenSSL libraries. A better resolution would be to change the global defaults so that the SCCM client does not need to be modified and uses the systems OpenSSL libraries.

Symptoms:

• CCMEXEC service will not stay running.
  • [root@system ~]# service ccmexecd status
    Checking for service ccmexec ccmexec.bin is stopped
• Centrify is installed on the system.
• In the scxcm.log
  • You see that the client begins to run and discovers the management points
  • There are no errors and may only be a couple of warnings about STATEID
  • The log files ends without error
• In the /var/log/messages file the following errors are shown
  • kernel: ccmexec.bin[xxxx]: segfault at xxxxxx ip xxxxxxx sp xxxxxxx error 15
  • kernel: ccmexec.bin[xxxx]: general protection ip:xxxxxxx sp:xxxxxxxxx error:0 in libcrypto.so.1.0.1e[xxxxxxxxxxxxxx]

Troubleshooting:

To determine if Centrify re-configured OpenSSL run the following commands.

• [root@system ~]# locate libcrypto
  /opt/microsoft/omi/lib/libcrypto.so.0.9.8
  /usr/lib64/.libcrypto.so.1.0.1e.hmac
  /usr/lib64/.libcrypto.so.10.hmac
  /usr/lib64/libcrypto.so.1.0.1e
  /usr/lib64/libcrypto.so.10
  /usr/share/centrifydc/lib/libcrypto.so
  /usr/share/centrifydc/lib/libcrypto.so.0.9.8

You will see that there is a newer version of libcrypto library in the /usr/lib64/ directory compared to the ones in the Microsoft and Centrify directories.

• [root@system ~]# rpm –qa openssl
  openssl-1.0.1e-30

This shows the version of OpenSSL that the OS originally installed.

• [root@system ~]# openssl version
  OpenSSL 0.9.8w-fips

This shows that the currently registered version of OpenSSL is the one that Centrify installs and overrides the version that was originally installed on the system.

• [root@system ~]# ls -lh /opt/microsoft/omi/lib
  libcrypto.so.0.9.8 -> /usr/lib64/libcrypto.so
  libssl.so.0.9.8 -> /usr/lib64/libssl.so

This will show all the files in the directory. Notice that the libcrypto.so.0.9.8 and libssl.so.0.9.8 files are links to the library files in the /usr/lib64/ directory (the system’s OpenSSL files).

• [root@system ~]# find / -name "libcrypto*" -exec ls -l {} \;
  /opt/microsoft/omi/lib/libcrypto.so.0.9.8 -> /usr/lib64/libcrypto.so /usr/lib64/libcrypto.so -> libcrypto.so.1.0.1e /usr/lib64/libcrypto.so.1.0.1e /usr/share/centrifydc/lib64/libcrypto.so -> libcrypto.so.0.9.8 /usr/share/centrifydc/lib64/libcrypto.so.0.9.8

This shows all the libcrypto files on the system. It shows that the Microsoft libcrypto file is linked to the /usr/lib64/libcrypto.so file and that file is linked to the libcrypto.so.1.0.1e library file. It also shows the Centrify libcrypto files are linked to its own version of the libcrypto file (the version that is Centrify registers with the system and overrides the original version of OpenSSL that was installed).

Resolution:

Now this is a problem because the OS has OpenSSL version 0.9.8 registered but the Microsoft SCCM client is linked to libraries associated with OpenSSL version 1.x. In order to fix this the links to the libcrypto and libssl files need to be changed to point to the Centrify version.

NOTE: The better way to fix this would be to change the global defaults to that the registered version of OpenSSL is the one that originally came with the system.

NOTE: This issue will not be noticed if the OS originally had OpenSSL 0.9.8 installed. Such as CentOS 5 or RHEL 5. I have only seen this issue with CentOS 6.x+ or RHEL 6.x+.

• Verify that the CCMEXEC service is stopped.
  • [root@system ~]# service ccmexecd status
    Checking for service ccmexec ccmexec.bin is stopped
• Change the ln (link) reference for both libssl and libcrypto in the /opt/microsoft/omi/lib/ directory.
  • [root@system ~]# cd /opt/microsoft/omi/lib/
    [root@system lib]# rm libssl.so.0.9.8 rm: remove symbolic link `libssl.so.0.9.8'? y [root@system lib]# rm libcrypto.so.0.9.8 rm: remove symbolic link `libcrypto.so.0.9.8'? y
    [root@system lib]# ln -s /usr/share/centrifydc/lib64/libcrypto.so libcrypto.so.0.9.8 [root@system lib]# ln -s /usr/share/centrifydc/lib64/libssl.so libssl.so.0.9.8
• Start the CCMEXEC service
  [root@system lib]# service ccmexecd start
  Starting ccmexec: Starting ccmexec: [ OK ]
  [root@system lib]# Logging Configuration File for SCX CM::/opt/microsoft/configmgr/etc/scxcm.conf
• Check to see if the CCMEXEC service is running of if it stopped
  [root@system lib]# service ccmexecd status
  Checking for service ccmexec ccmexec.bin (pid xxxx) is running...