Protecting a BIT of Integrity BYTES

Leveraging the NIST Cybersecurity Framework to apply necessary albeit painful and often overlooked cyber changes to protect your most critical high-value assets (“Crown Jewels”) from advanced cyber threats

The National Institute of Standards and Technology (NIST) has a comprehensive Framework for Improving Critical Infrastructure Cybersecurity aka the Cybersecurity Framework summarized as:

·       Identify high-value assets·       Protect against known and unknown threats·       Detect attacks and anomalous activities·       Respond to suspicious activity·       Recover from breach

This post will focus primarily on Identify function's Asset Management component and the Protect function of the NIST framework as it relates to often overlooked operational changes needed to isolate critical high-value assets.

What to Protect?

Keeping our systems free from corruption and out of control of unauthorized actors is difficult; but often we never even identify what really needs to be protected. Like the real Crown Jewels are locked in bullet-proof glass under watched guard in the Tower of London; there are certain high-value assets in every organization that need to be identified, classified according to risk, and protected appropriate in some cases without allowing exposure to untrusted masses.

Asset Management (Identify)

The data, personnel, devices, systems, and facilities that enable the organization to achieve business purposes are identified and managed consistent with their relative importance to business objectives and the organizations' risk strategy – NIST Cybersecurity Framework

Intruders often know more about the assets our network than the IT admins managing the systems. Here are some of the often overlooked items that affect control of or access to critical assets:

Privilege Credentials - with administrative control over high-value assets. Many IT admins and cyber teams can’t even identify high privilege accounts and security groups outside of the most basic well-known users and groups other than SQL DBA/sa account, members of Domain Admins and Enterprise Admins, and the all-powerful Linux root and Windows Administrator account. Because of this most organizations don’t really inventory or know whom have access to these shared high privilege accounts, and therefore don’t review the audit the use of those accounts. There are security groups that can give control of AD that may not be intuitive.

For example: Some non-intuitive built-in AD groups can become Domain Admin equivalent:

• Backup Operators – Can back up and restore all files on DCs
• Server Operators – Can log on interactively, create and delete shared resources, start and stop some services, back up and restore files, format the hard disk of DCs
• Group Policy Creator Owners – can modify Group Policy that affect DCs
• Print Operators – can load and unload device drivers on all DCs

Warning! The advanced intruders know how to exploit these capabilities and elevate privileges within Active Directory, but IT admins and cyber teams don’t usually know how to safely delegate these to prevent escalation of privilege.

See: https://aka.ms/tier0 for a list of creds that can get direct control of enterprise identities.

Cleartext Passwords – Many IT admins and Cyber teams don’t scan and inventory their networks for cleartext passwords stored in configuration files, text files, applications located on network shares, in Group Policy Objects and/or in databases that are readable by large groups such as “Authenticated Users” or “Everyone” security groups. PowerShell scripts that can be leveraged for finding passwords on file shares without requiring expensive tools, and simply understanding where these are in GPOs and applications should be the responsibility of the IT admins and cybersecurity teams applied

See: https://github.com/thechrisharrod (coming soon) for a PowerShell script hunting for cleartext passwords on the network.

Note: I will update the link to the exact script above once it uploaded

Sensitive Data (e.g. Personally Identifiable Information (PII) and Financials) – Many times the IT admins and Cyber teams do not understand where their sensitive data is stored, how it is transmitted, who has access, or what systems processing and storing that data. Often these have additional security regulations governing the usage and spillage of this data which needs to be clearly identified as to which regulations apply (e.g. HIPAA, GLBA) and additional security controls that need to be applied.

Advanced threat actors scan the networks looking for data. IT admins and cyber teams need to scan their networks to see where data has been duplicated, copied or archived.

• Decommissioned Systems - If a files server has been migrated, make sure old server is properly decommissioned and taken offline as sometimes old servers stay online for years unmanaged and exposed.
• Backups Data - Make sure backup tapes are monitored and tracked as well as any system where backups are copied to in any unencrypted format
• Private Keys - Make sure private keys for certificates are deleted after installation on web servers as often PCKS#12 files are copied to shares with a .txt file with the password. This is a treasure for bad guys who scan for these and steal certificates quietly for man-in-the-middle attacks.

See: The Microsoft Compliance site for current information on identifying applicable standards and regulatory compliance and https://github.com/thechrisharrod (coming soon ) for a PowerShell script finding data that low privileged users on the network (e.g. members of “Everyone” or “Authenticated Users”) can access and/or modify.

Note: I will update the link to the exact script above once it uploaded

• Control Systems (e.g. utilities, medical equipment, weapons) – Many IT admins and cyber teams do not understand the implications of losing control of their Supervisory Control and Data Acquisition (SCADA), Process Control Networks (PCN), and Industrial Control Systems (ICS) which often have direct control of equipment that can affect the environment, public safety and health systems
• Point of Sale (POS) and Automated Teller Machines (ATM) also fall under a similar category whereby access to these can lead to theft of finances and ultimately identity theft.

Warning! Usually these  systems have absolutely no business need to be on the same networks used by the general employees nor or managed from systems on the Internet or checking e-mail. These systems should be inventoried and categorized based on their potential risk if compromised. In most cases, these should be in an enclave with dedicated admin workstations with no access to the general user networks.

See: Privileged Access Workstations (PAW) for ways to secure access to these systems and Server and Domain Isolation for methods to isolate the systems from the production general use network.

Critical Computers (PCs and/or Servers) that manage any of these high-value asset (e.g. Domain Controller) need to be better protected than general systems. Many IT admins and cyber teams don’t even know all the systems (network devices, PCs and servers) are on their network or who manages them. These rouge systems often go unmanaged and unpatched for months or years and become a security risk to the production network, but fear of causing someone downtime causes them to be left online and unmonitored.

Recommendation: Scan regularly for unknown IP addresses of systems connecting to critical computers and audit the security logs for successful and failed logons. Keep accurate and up-to-date inventory of all systems on the network and alert on and investigate rogue or unknown systems and IP addresses. Patch or turn off ALL systems including those that are of unknown origin. If system is in production and is shutdown, someone will either claim ownership or you can schedule the system for decommission and removal.

See: SCCM Hardware Inventory and Hardware Inventory for Linux and UNIX 

• Critical Software (e.g. security and management tools) that grant full admin control over critical high-value assets – Most IT admins and cyber teams do not understand the risks associated with software agents and services (Microsoft or 3^rd party) installed on high value asset machines. Control from lower assurance PCs to systems managing Tier-0 accounts (e.g. Certificate Authorities, Domain Controllers), high value data servers.

Warning!  A lack of end-to-end understanding of these security implications that increase attack surface and grant command and control functionality to unauthorized individuals usually and often leads to a false understanding of the security profile of many systems.

For Example: Patch management systems (e.g. SCCM), VMWare/Hyper-V consoles, SAN tools, and/or antivirus systems controlling domain controllers can give complete control of the entire forest to those admins. Users that can control those agents or services on the DCs from low assurance PCs on an "assumed breach" network unnecessarily give advanced attackers full control of the network and the “crown jewels” seamlessly.

See: SCCM Software Inventory and https://aka.ms/tier0

• Outgoing and Incoming Trusts – Many organizations have way forest exposed an organization to security risks. All of those trusts need to be understood, well documented and monitored by the security organizations in the agencies. When possible Selective Authentication trusts or leverage Active Directory Federation Services (ADFS) in lieu of trusts.

See: Security Consideration for Trusts for threat scenarios to trusts made between two forests

How to Protect?

Once these are inventoried and understood; how do you then protect the integrity of those BITS and the systems on the network controlling them?

Access Control (Protect)

Access to assets and associated facilities is limited to authorized users, processes, or devices, and to authorized activities and transactions – NIST Cybersecurity Framework

Controlling access seems like a simple concept, but it should be more than security access control lists and security group access for critical high-value assets. Microsoft has published two whitepapers and numerous documents on defining the Tier-0 accounts that can grant complete control of Active Directory (AD) and all assets in the environment and how to deal with credential theft and Pass-the-Hash including the following public guidance and consulting protection services offerings.

Isolate High-Value Assets – Isolation of Tier-0 credentials and systems or data identified as high-value assets via IPsec or other security mechanisms is highly recommended to preserve data integrity and prevent unauthorized command and control (C&C) of a data’s host. However, such thinking is counter intuitive for most admins and is a difficult challenge often met with resistance to change. Rarely do IT admins use an “assume breach” approach to preventing a lower security asset such as their Internet-connected PCs from connecting to high-value assets for administration. Allowing that is a violation of any hope for integrity of the critical assets that need high-assurance protection.

Note: Designing an isolated network from scratch requires a departure from traditional thinking about operations and modern information technology (IT) protection methodologies. Isolating critical assets in an existing operational network while retaining availability of data systems and functional utility of its applications is even more unthinkable for the traditional network or systems admin, IT manager, or even a chief information officer (CIO) who focus on productivity, cost and performance while relying on security tools and access control lists for protections.

We must adapt to the advanced persistent threat (APT) with focused agility. Recently, an article describing micro-segmentation was published as a new technique to divide the “physical networks into thousands of logical micro-segments using software-defined identities” See the Computer Weekly news report on June 24, 2016. Is this a new idea? Not really. Microsoft has been promoting Server and Domain Isolation using IPsec and Group Policy for more than a decade since 2005; but more must be done.

Dedicated Admin Workstation – It is imperative to use dedicated admin workstations aka Privileged Access Workstations (PAW) to provide a dedicated operating systems for sensitive tasks protected from Internet-based attacks and threat vectors. Separating these sensitive tasks and accounts from the daily-use workstations and devices provides very strong protection from phishing attacks, application and OS vulnerabilities, various impersonation attacks, and credential theft attacks such as keystroke logging, Pass-the-Hash, and Pass-The-Ticket.

Dedicated admin workstations, take you from an at-risk environment where a compromised PC from a phishing attack leads to control of a privileged server such as this,

to an isolated admin environment such as this where the admin workstation has no access to the Internet, e-mail or even other general use PCs on the network:

Enhanced Security Administrative Environment (ESAE) – For a more comprehensive solution, leverage Microsoft Cybersecurity Services to deploy a full Enhanced Security Administrative Environment (ESAE). ESAE is highly recommended and a much more comprehensive solution to help thwart critical elements of credential theft attacks by limiting exposure of administrative Tier-0 credentials anywhere in AD or on the network including via 3^rd party applications (e.g. services, agents on  DCs). This changes the entire operating environment to isolate highest privileges from Web facing systems and general use computers.

ESAE deployment utilizes:

• Tier classifications for all users (e.g. Interactive Logons, Certs, Services)
• Tier classifications for all computers (e.g. Windows, Linux, Macintosh)
• Dedicated administrative AD forest protecting Tier-0 assets leveraging
  • IPsec isolation
  • One-way selective authentication trusts
  • Dedicated admin workstations for Tier-0 only enhanced security configuration and IPsec isolation from the production
  • Smart Card required for interactive (SCRIL) logons only for Tier-0
  • Rotation of SCRIL on a frequent basis for new random password hash rotation
• Credential hygiene and operational best practices (e.g. not exposing credentials across tiers)

Securing Lateral Account Movement (SLAM) – Leverage Microsoft cybersecurity services for Securing Lateral Account Movement (SLAM) to help networks defend against lateral traversal attacks by using:

• GPOs to ensure that local system accounts are not allowed to connect to other client systems
• Local firewall on client machines to ensure they do not have connectivity to each other unless specifically allowed
• Local Administrator Password Solution (LAPS) leveraged to randomize the local administrator passwords and storing passwords in AD in a secure fashion

Awareness and Training (Protect)

The organization’s personnel and partners are provided cybersecurity awareness education and are adequately trained to perform their information security-related duties and responsibilities consistent with related policies, procedures, and agreements – NIST Cybersecurity Framework

In most environments, the IT dmins have a good understanding of security configuration of tools, internal polices, and they understand basics of identity systems and active directory security but have little or no understanding of how advanced threat actors attack an environment. This is a shortcoming. On the other hand, cyber teams have a really deep understanding of attack methodologies, but have little to know understanding of why persistence mechanisms that involve obscure security changes in Windows/Linux Operating Systems or AD security matter (e.g. access to Linux keytab file or exploiting AD’s AdminSDHolder).

Traditionally there has little feedback and/or cooperation between the teams and thus a cyber-attack may go unnoticed for years when no malware is being used for persistence. A persistence mechanism such as modifying a file system or registry such as replacing or debugging sticky keys to gain access to Windows or modifying an attribute for Sneaky Active Directory Persistence Tricks such as exploiting AdminSDHolder to come back in often are not well understood when there is no malware in place and no shiny object tool to explain how the default security has been circumvented.

Data Security, Information Protection (Protect)

Information and records (data) are managed consistent with the organization’s risk strategy to protect the confidentiality, integrity, and availability of information – NIST Cybersecurity Framework

Encrypt and secure access to only those that need access.  That’s not problematic, but over time permissions may drift, and data that should not be accessible to everyone becomes available. For example, you can run PowerShell regularly and see what is accessible from low-privileged users.  For example, (also from thechrisharrod) as a quick sample of a PowerShell script to see to what shares a user would have access.

See: https://github.com/thechrisharrod (coming soon) for a more comprehensive PowerShell script for finding data that low privileged users on the network (e.g. members of “Everyone” or “Authenticated Users”) can access and/or modify.

Note: I will update the link above to the exact script above once it uploaded

Processes and Procedures (Protect)

Security policies (that address purpose, scope, roles, responsibilities, management commitment, and coordination among organizational entities), processes, and procedures are maintained and used to manage protection of information systems and assets – NIST Cybersecurity Framework

Change the way you build to protect your Supply Chain

Mitigate Supply Chain Risks Using Known Good Media (KGM) – Mandiant recently published their own version of a cyber trends report describing how threats recently compromised image repositories for Cisco router images on file shares that were then loaded on production devices creating evil routers.  This should not be a surprise!

For many years BIOS, O/S, and application vendors have posted image hash files on their public web sites or signed their own code to ensure anything downloaded retains its integrity. What IT admin ever checks these hashes when the downloads install properly albeit with malicious add-ons?

Microsoft addresses software integrity its Risk-Based Approach to Managing Software Integrity whitepaper and the Cyber Supply Chain Risk Management: Toward a Global Vision of Transparency and Trust  whitepaper published in 2011. You must validate the integrity of software if you want to ensure we are deploying secure software.

There are specific known good media processes that can be followed to ensure using secure hashing algorithms (e.g. SHA-2 suite) and/or signed code is used to verify that those images were not tampered with or modified (potentially maliciously for persistence after an attack). A simply start would be to validate the hashes and/or signatures of all software (e.g. .ISO, .IMG, .EXE, .BIN) installed in an environment during any build/deployment and to monitor the hashes on image repositories .

See: Enhanced Security Administrative Environment (ESAE)

Cyber Operations and Credential Hygiene – “ Assume Breach” and securing privilege access should be put in practice. In reality the idea of operationalized security with an assumed breach mindset at least for Internet facing PCs is almost non-existent in most organizations that have not gone through a major incident response.

Any of the highest privilege accounts that can give complete control over the entire identity system and thereby all data and control systems should never be exposed to a lower tier networks and data systems. UNIX/Linux files such as Passwd/Shadow and Keytab files should be secured by root and the root password randomized as well. Cached Kerberos tickets (klist) should also be secured properly.

Local Administrative accounts on all systems should be randomized using tools that many organizations already own such as: CyberArk, Lieberman Enterprise Random Password Manager (ERPM), Xceedium. Even without such privileged account management tools, AD users can leverage Microsoft’s free Local Administrator Password Solution (LAPS) solution for randomizing local server and workstation administrator accounts.

See: LAPS and Securing Privileged Access.

Maintenance (Protect)

Maintenance and repairs of industrial control and information system components is performed consistent with policies and procedures – NIST Cybersecurity Framework

Patch management should be no-brainers but apparently it is not.

Patch Management – Many IT admins and cyber teams patch their major operating systems (Windows, Linux/UNIX, AS/400) and most commercial off the shelf software (COTS) regularly that re running on servers and PCs in the production forests/domains. However, that is not often the case when it comes to custom in-house developed software, voice over IP (VOIP) telephony systems, old mainframe apps, thin clients (often assumed secure), out of band or standalone systems in enclaves, and test labs. Patch management systems only report on the systems they manage, so a general lack of understanding what is on the network is problematic as well.

There have been numerous cases in the past few years where well-crafted voicemails sent to a VOIP phone system will allow complete takeover of the phone systems or a thin or zero client can be exploited; but many times the IT admins and cyber teams do not look at the phones or thin client hardware and software. IT often do not understand the security implications of these systems because IT thinks the telephony teams manage security and telephony teams believe the server teams manage the security of VOIP systems. For thin clients, IT systems sometimes assume the attack surface is low or impenetrable and thin or zero clients don’t even need patching. This is not true and there are numerous examples of exploits on these.

Warning! Patch everything, everywhere and never assume something doesn’t need to be patched. Subscribe to U.S. Cert alerts and all vendor specific vulnerability notification systems for alerts.

See: Security Development Lifecycle (SDL) for best practices for security development and patching in-house develop software and review the components of a well-managed patch management program using System Center.

Truly Offline / Un-writable Backups – To prevent the catastrophe and risk of a fully unrecoverable destructive attack, you must have some form of offline backups. Many organizations use online Storage Area Network (SAN) disk backups and cloud services archives to save money over offline tape systems without consideration for a destructive cyber-attack. There have been numerous wide-scale destructive attacks that completely wipe out all online backups before over-writing all production data in a full wipe or encryption attack.

While online backups are great for natural or man-made disasters to equipment and networks, they are not designed to withstand an attack by a determined adversary with backup operator and/or enterprise admin credentials that can destroy both the backups and the online systems so that there is no recovery .

For Example: If the domain controller backs up to disk and the domain admin credentials are stolen in a Pass -the-Hash attack (very common), then the attacker could wipe out the backup media, then flatten the network and there is no recovery for Active Directory or any data that doesn’t have a tape backup turning a simple restore into an unrecoverable disaster.

See: Crilock Recovery and Ransomware Risks

Protective Technology (Protect)

Technical security solutions are managed to ensure the security and resilience of systems and assets, consistent with related policies, procedures, and agreements – NIST Cybersecurity Framework

This includes protecting communications and control networks, enforcing least privileges, protecting removable media, and reviewing of audit logs. While this is great, do not rely on “Shiny Objects” or any or the latest and greatest security Tools. Everyone has security tools, everyone gets hacked at some level, not everyone knows when or where they are compromised as most IT admins do not have the staffing resources or spend the cycles to review their audit logs unless actively troubleshooting a known issue.

Cyber issues are routinely not known about however.

Weak Enclave Topology – Many times IT admins and cyber teams create a semi-isolated or weak “enclave” that is accessible from a production network over a secure connection to a regular workstation or to PC simply by accessing a jump server or VPN connection. The credentials authenticating to the enclave then are only as protected as the “assumed breach” PC and often web data is even cached locally on the production network workstation making the workstation just another hop to the enclave for the bad guys.

This is especially problematic when the enclave rules do not actively address the threats of credential theft and lateral traversal. IP restrictions will not cut it. This threat needs to be clear and there are many environments where enclaves should really be a separate and completely inaccessible from the production network. SCADA and ICS are good case in point examples that need dedicated PAWs or completely separate admin environments.

Whitelisting – Whitelisting should be used IT admins and cyber teams more regularly. NIST has very good guidance on whitelisting. The NSA has published a very good free whitepaper guidance for deploying the Microsoft’s free whitelisting tool, AppLocker that is titled, Application Whitelisting using Microsoft AppLocker. Other than compatibility testing, there is no reason every organization that owns a commercial whitelisting technology should not start deploying it or the free AppLocker.

See: Application Whitelisting Using Microsoft AppLocker  and AppLocker Guidance on GitHub.

User and Entity Behavior Analytics (UEBA) –  Gartner published a whitepaper on using User and Entity Behavior Analytics (UEBI) to spot adversaries early.This falls under the detect function of the NIST framework, but there needs to be at least a deployment of these tools and a basic configuration to collect data for behavior analysis.

Security Log Configuration and WEF – Most organizations own great tools to collect event logs, SQL logs, IIS logs, Apache logs, Sysmon logs, WiFI logs and VPN logs among others. Most admins have access to a security incident and event management (SIEM) solution (e.g. ArcSight, Splunk). Windows also has built-in free functionality of Windows Event Forwarding that can be used for free.

These UEBA, SIEM solutions and WEF need to be configured in order to capture relevant data for realtime and forensic analysis (e.g. capturing successful and failed logons from legitimate user accounts). Most advanced attack in the last half decade has used some form of credential theft and once inside an APTs or an insider threats simply use legitimate credentials often to move laterally move in a network and go unnoticed for months or years without the use of malware.

The NSA has published an excellent whitepaper on Spotting the Adversary with Windows Event Log Monitoring that is a great resource for understanding events that need to be monitored. There are additional tools that many organizations already own from a licensing standpoint but have not deployed the capabilities. This includes Microsoft Advanced Threat Analytics which uses deterministic rules and machine learning to analyze the captured data as a UEBA solution.

See: Microsoft ATA  and NSA’s Spotting the Adversary whitepaper

Conclusion

Rise above the threats! Leverage the NIST Cybersecurity Framework and follow best practices to isolate and protect your most critical “crown jewels” and tier-0 credentials using operational security practices and not just dependence on the latest “shiny object” security tools. Bad guys have these same security tools before they attack, so we need to change the way we isolate and operate on our network. These changes can be painful and often not intuitive, but defending against advanced attackers require advanced operational defenses to keep a breached PC from becoming a totally owned network.