Creating a standard patching process

The following points should be considered when creating a standard patching process.

Understanding the patch

First, know what the patch is for. Read the patch notes and details from the vendor and understand:

Getting the patch files

When you get the patch files, confirm that the files come from an appropriate source. Before downloading files from a vendor’s website, check the TLS/SSL certificate details and verify that the owner of the domain is the owner you expect it to be. Phishing attacks use spoofed websites or emails to convince users to download and run malicious files. Don’t download your patch files from sources that you don’t share an encrypted channel with - like a website served over HTTP or a file share over FTP.

Often patches are made available through mirrors, or servers that have copies of patch files that may be geographically closer to your server (which can help with download speeds). You should only use mirrors that have signed patch files, so that you know it has from the right and trusted source, or at a minimum a checksum hash.

After you download the file, run a checksum (like a hash check using a hash function from the SHA-2 family) over the file. A vendor should provide you with their hash. Compare the two to make sure you have the right file. If the checksums are not the same, there could be a problem with the integrity of the file. You should contact the vendor to check.

TIP: Download your patch files to a central location within your environment (after you have validated the source). For example, use Microsoft’s Windows Server Update Service (WSUS) to download all your Windows software (including operating systems, web browsers, and other applications). This will save you time in your patch change window and help you later when you move to automate the process.

Patch window and timing

Finding the right time to apply updates is a balancing act. You need to find the right balance of ensuring minimal impact to service (consider how you can patch without disrupting service at all), while also minimising disruption to your staff, and their personal time. Setting a patching window during a low traffic period will reduce the impact to end users. These windows should be long enough to allow the patching to finish. Consider the following things when setting the patching window:

Backup and rollback

Any time you are making changes to a system, it is important to have a rollback plan. This may involve taking backups before applying patches. Sometimes even after testing a patch successfully, there may be some bugs that are discovered later that disrupt your organisation. Having a backup to roll back to and having a clear way to make the call to roll back to a backup are important. The decision to roll back should consider:

Testing

First deploy a patch to a test environment or to a single instance before it is deployed to your entire environment. Test to make sure that the critical functions the software provides, still operates the same way it did before. This could be patching a gateway and testing to make sure that traffic still flows correctly, or patching a business application and testing to make sure the key features still work and produce the same outcomes. You can’t always test everything - it is usually a best effort attempt.

TIP: Vendors put their patches through their own code development and testing process before they are released. When assessing the level of testing that you need to perform before applying a patch to a live environment, consider the amount of testing that vendors perform before releasing patches to the public and the level of risk posed by the software if left unpatched with a known vulnerability.

Notification to end users

Patching should be a regular process and most end users don’t need to know when it happens. To reduce notification fatigue, you should only push notifications to end users when emergency patches are applied outside of the regular patch window or when there is an action required on their part (ex: a system reboot). To encourage a culture of patching, it would be beneficial to have information and guidelines available to end users so they can learn more about how the process works, how the impact to end users has been considered (with areas like patch windows and test scope), and what vulnerabilities were recently patched.

Verification

After the patch has been deployed across the environment, it is important to verify that patching is complete. You could have received the notification that the patch file was pushed to a server, and the file was executed - however the server could have failed to reboot and the patch may not be in effect just yet. If the patch you deployed was widespread throughout the environment, you can perform this check by using active scanning tools to see if the vulnerabilities have been resolved or if the software has an updated version number.

Emergency

When a patch is released with no prior warning and outside of a normal vendor release cycle - that usually means it is critical. You should be prepared to review these type of emergency patches and determine if it is applicable to your environment. If the answer is yes, you should take the patching process that you have now and make changes to it that cuts down the time to deploy and adds to end user notification. Depending on the severity, you may decide to limited testing and instead notify end users of the severity of the issue and thank them for understanding that their use of the software may be interrupted while patches are rolled out.

TIP: The steps above are not just unique to patches - these are typically steps that are followed for other types of changes too. When deciding on a patch process, you can always use your change management process as a foundation - just make sure to include the steps that are somewhat unique to patching (like validating file checksums).