The rapid rise in remote work, cybercrime, and the decentralization of devices has caused Microsoft to rework their patch management process. Every company who uses Microsoft-based patching will be affected by this change.
The new enforcement model gives less control to individual IT departments for deploying patches in their own time and way. On the other hand, Microsoft’s new process is resulting in increased security and more efficient deployment of patches. Benefits you can expect to gain once your organization gets the process down.
In this series we’ll explain the new Microsoft patch management model as well as:
- Tools you can use to put the model in place at your organization
- How to configure these tools
- Implementing patching best practices
- How third-party patching fits in
The first post of our series will go over the importance of patching and challenges to patching efficiently. We’ll also cover some global changes Microsoft made to their patch management process and how those changes will affect your patching moving forward.
What Is Patch Management?
Chances are, you’ve been patching systems since you brought the first device into your organization. Simply put, patch management is the process of controlling the deployment of software and operating systems updates to devices within your organization. This includes:
- Obtaining and evaluating patches
- Deploying approved patches
- Verifying deployments
IT patch management teams are typically devoted to performing the patching process in larger environments and environments where systems need to be carefully managed.
Patching Challenges
Patching the operating system and the applications that run on your computer presents several challenges to organizations, including:
- Obtaining patches for each operating system and application running in the environment
- Testing patches for potential flaws
- Deploying patches effectively
- Monitoring to make sure the patches have successfully installed
Organizations have been trying to determine the most effective means of keeping systems up to date for decades. For some organizations, patching everything as soon as the updates are available is the best option. Others take a more conservative approach, waiting months before applying the updates to machines.
Rolling out patches as soon as they are available is an option for organizations without a lot of proprietary software requiring testing or systems providing specialized functionality (machine automation, security system controls, or systems that require high availability).
Most user’s workstations do not fall into these categories, making them prime candidates for having patches rolled out quickly.
If the user’s system includes specialized software to access a service or application, patches should be thoroughly tested prior to updating.
Rolling out patches to several thousand workstations introduces its own set of challenges. Administrators need to monitor the rollout and validate that patching was successful. Without an automation tool or monitoring systems, administrators must manually apply patches or rely on end users to perform the patching on their devices. This is a less than optimal patch management solution and one that most large organizations cannot afford to implement.
At the same time, evaluating every patch as it is made available can be cumbersome. There are some months where hundreds of updates are released, generating a heavy administrative load. Smaller teams may not have the manpower to evaluate each patch individually so mass testing of updates becomes the norm. If issues arise during the testing phase the team needs to troubleshoot which patch caused the problem.
These challenges are intensified with the rise of decentralized devices and remote work, which have changed the landscape of our work environments.
When workers were in-house with their workstations attached to the corporate network, applying patches was performed over high-speed network connections. Removed from the corporate network, users are constrained by the speeds imposed by their internet provider and the VPN connection back to the corporate network. Organizations have found that they either need to improve their external network speed to accommodate the VPN bandwidth requirements, or they need to stagger their patching efforts over the course of several days. [WB1] [WB2]
With everything at stake for cybersecurity (see statistics section below), keeping patches up to date is threatening to become a more complicated and resource-consuming process every year.
Why Manage Patching?
Efficient software patch management mitigates many risks in your environment. For example, if you have a computer that is running old software or an older version of the operating system, the software may be full of bugs and can cause your computer to crash. Viruses, malware, and spyware can cause problems, especially with vulnerable systems that are not up-to-date on their patches. When these problems occur they can lead to issues like slow computers, constant crashes, blank screens, blue screens of death (BSOD), and much more.
One of the most infamous ransomware attacks, WannaCry, occurred in 2017. The National Security Agency discovered a vulnerability in the Windows operating system called EternalBlue. Microsoft quickly developed and distributed a patch for EternalBlue that became available in March of 2017. However, in May of 2107, WannaCry was released and started encrypting data on devices.
Many companies had not patched their Windows devices and were susceptible to WannaCry. It is estimated that over 200,000 devices were infected. Having a patch management plan that keeps systems up-to-date could have alleviated most of the infections. But attackers take solace in the fact that many companies and individuals do not keep their systems fully patched.
Fortunately, there are several ways to prevent scenarios like this from occurring in the first place. The first and most important is to install updates on your computer immediately after they become available. This ensures that you are always up-to-date on the latest security patches and fixes.
Next, make sure that all of the software on your computer is up-to-date. This includes the operating system and any programs that you use regularly.
There are several ways developers discover issues within their software. Sometimes it is through support calls when someone is unable to perform a function or they are receiving incorrect results when trying to perform an action. Other times, someone discovers that there is a security vulnerability.
With all of these scenarios, the developer needs to rectify the issue quickly. Organizations can’t use software that does not work correctly or run the risk of a security breach.[WB3]
Patch Management Statistics: What’s At Stake
Speaking of threats and vulnerabilities to unpatched systems or systems that are not up-to-date on their patches, here is some data to think about:
- 20% of vulnerabilities left exposed due to missing patches are classified as High Risk or Critical
- Most of the breaches that Microsoft has investigated could have been prevented by updating devices with the latest patches and operating systems. The Digital Defense Reports published by Microsoft provide details.
- Malware and Ransomware that have caused nightmares for companies, such as WannaCry and NotPetya could have been avoided by patching software on time.
- Around 80% of companies that experienced a breach or failed a security audit could have avoided their issues by keeping their operating systems and software patched.
In short, the more up-to-date your devices are, the more secure you will be. We get it – easier said than done, right? But being diligent about patching can keep your organization in a much better position. It’s worth investing the time and budget to deploy a quality patch management solution that will prevent a significantly greater cost of lost productivity for your team, or worse, a data breach.
Legacy Patch Management
Depending on the complexity of the organization, the patch management process can range from basic to very complex. There are many small companies that have traditionally updated all of their systems manually. They don’t have a large number of machines and typically have a very limited budget so an automated patching process is not something they will have invested in.
Instead, they instruct users to patch their devices when they see an update is available. The technical staff is responsible for updating servers, firewalls, and other company infrastructure.
Now consider for a moment just how many patches are needed in an enterprise environment. Operating systems and typical desktop apps will need patching of course. But so will servers, routers, firewalls, and other pieces of network infrastructure, as well as websites and website apps. Laptops and mobile devices, too, need consistent patching.
Very quickly, the number of patches required on a consistent basis becomes unmanageable.
Installing a patch also requires a process to guarantee success. It’s not unheard-of for a faulty patch to prevent a critical application from functioning. When this happens to a single end-consumer, it’s frustrating. Now imagine it happening to your entire finance department.
These scenarios make testing and deploying patches a critically important part of patch management. In a Microsoft-based patching scenario, there have traditionally been two tools used to manage patching:
- Windows Server Update Services (WSUS)
- System Center Configuration Manager (MCM/SCCM) (now known as Microsoft Configuration Manager (MCM))
WSUS is used in both of these patching scenarios. SMCM/SCCM & MCM use WSUS as their patch subsystem, while having better control over the deployment of the patches. No matter which of the tools is used, there is a lot of configuration that goes into automating the patching process.
But these solutions don’t run themselves. Once configured, your administrative team will need to make sure they are diligent about updating the processes, and making sure all the required patch types are being managed and the processes are providing the correct level of support.
Microsoft Patch Management
Microsoft provides updates through their Windows Updates service to millions of devices, free of charge. As long as a device has internet access, these updates are available to install as soon as Microsoft releases them. For the IT patch management team at an organization, leveraging these updates from Microsoft can save a lot of money and time. The key for any organization that wishes to leverage Windows Updates is controlling when the updates are applied to their endpoints.
By tying Microsoft Updates to your patching process and leveraging Intune and Microsoft Configuration Manager to deploy third-party updates to your devices, you can alleviate many of the deployment challenges that administrators have. Together, you can deploy updates within your network without incurring heavy internet download costs and update the remote devices without having to saturate VPN connections.
Microsoft’s Patch Management Is Changing
Microsoft has been in the process of updating their patching process. Why?
They found that many devices were not keeping up with the patches required to keep devices safe. As with the case of WannaCry, if companies had applied the March 2017 patch when it was released WannaCry would not have been a big deal in late May of that year.
Also, updating devices from one version of the operating system to the latest version can be problematic if the device is not up-to-date.
Newer, more reliable versions of Windows and other software take longer to upgrade when the operating system or application is dated. If you were to look at the size of the upgrade packages applied to devices, devices that are up-to-date on their patches have smaller overall patching requirements and update far faster.
Plus, patched machines are safer. Microsoft’s goal is to keep as many systems updated as they can. Setting what was once seen as an aggressive schedule for patching systems is proving to be more effective and efficient for companies. Windows 10 and Windows 11 are designed to be updated and upgraded far easier than any former version of Windows.[WB4]
How Will Microsoft Patch Management Change?
Moving patch management to the cloud is on everyone’s mind these days. While Microsoft has already positioned itself to provide patches from its Windows Updates service, third-party solutions are available to provide non-Microsoft updates for software and drivers from cloud sources. Some of these solutions leverage services such as Microsoft’s Intune or VMware’s Workspace One to deliver packaged updates to devices. Others provide their own cloud solution to deliver update packages to endpoints.
The key to these solutions is to enable administrators to quickly deploy updates with minimal administrative overhead. Instead of requiring administrators to test and validate every update, teams of professionals test updates in their labs using automation techniques to ensure the validity of the patch. They then package the applications for deployment, reducing the administrator’s workload when managing third-party patches. This enables your organization to quickly deploy the updates to devices with a faster turnaround time.
Staying Ahead of the Curve with Patch Management
The faster you update your systems, the less likely you will be affected by malicious actors. Most workstations and laptops used in organizations will benefit from rapid deployment technologies endorsed by Microsoft and their partners. They are committed to distributing reliable patches and updates for your devices.
But that doesn’t mean that the updates and patches are going to be 100% perfect. There may be some older applications or proprietary software that might not be compatible with certain updates. You should still be diligent about testing to make sure you will not adversely impact a large number of your users.
Microsoft’s New Patching Solutions
Microsoft has positioned WUfB as their premier update tool, setting it up to function effectively for both remote workers and in-house staff. There are two components used within this solution:
- Windows Updates for Business, which provides the updates.
- Delivery Optimization, which is used to reduce the Internet traffic when updates are downloaded to devices.
When both technologies are used together, devices start downloading and caching patches. After a few devices have the patches cached, other devices looking for patches will retrieve them from systems with the cached patch. Getting the patches from a local system reduces the overall Internet traffic load.
Modern Patch Management Processes
Microsoft is pushing towards an automated patch deployment solution centered around their new service, Windows Updates for Business (WUfB). From the lessons they have learned over the years and the best practices they have defined, Microsoft has created a solution that can automate the patching process.
Your patching team will still have control of how quickly patches are deployed to devices, but within a limited window of deployment opportunity. Users have some control over when the patches are applied to their systems, but safeguards are now in place so that users cannot defer the updates indefinitely. [WB5]
Update Rings
You have probably heard the term Update Ring if you have spoken with anyone responsible for patching. Simply put, an update ring is a group of computers that are updated at approximately the same time with the same settings. By having differing settings within each ring, you can control how the updates are delivered to devices and how users can interact and manage updates as they are delivered to their device.
As a part of our patch management service at Model Technology Solutions, we typically configure a minimum of three update rings for our clients. Usually, those three are all that are needed. The three rings are named something along the lines of:
- Test
- Pilot
- Production
Their names reflect the patching stages used to test patches and move them into a production environment.
The Test ring is used as the first evaluation of new patches that are released. This is usually a small group of machines that have configurations mimicking the devices that users work on. Very soon after patches are released, usually within one or two days, the device is instructed to download the updates.
Users are prompted to update the device or specify a time when they want the device to update. They are only granted a short window of opportunity to install the updates. If they don’t update the device it will automatically update at the deadline set by the patching team.
Administrators review the install and validate the patches did what they were supposed to do. If all looks good, they can allow the next ring to proceed with updating.
The Pilot ring is the next to receive the patches. The devices in the Pilot ring are representative of the devices used widely throughout the organization. Typically, the users who work on the Pilot devices are more tech savvy and are trusted to test the software on their machines to see if everything works after the patches are installed.
The Pilot Ring is configured to download and notify the users shortly after the deadline for the Test group has been completed. This is usually one week after the updates were released. Again, users will be able to defer their updates for a little while, but the updates will be enforced at the deadline if they don’t approve updating before then.
Once again, the patch management team will need to validate the deployment of the patches to make sure that nothing was adversely affected. If there are any problems with the Test or Pilot phases, administrators have the opportunity to pause the updates until they can determine what is causing problems with patching the devices. If everything appears good, the final phase can begin.
The final Production ring is used to roll out the patches to the remainder of the organization. By the time the patches reach this population of device, the patches have been vetted in-house as well as throughout the software community.
Peer Caching
Peer Caching is a technology that allows devices to store pieces of software and securely share that software with other systems. Microsoft’s version of Peer Caching is called Delivery Optimization. There are other vendors that provide similar tools that can be used with third-party patching services.
One of the benefits of Delivery Optimization is the efficient manner in which the updates are stored and shared. Every device that is identified as a peer can cache parts of the software shared with other devices. This means the software does not consume a lot of drive space on the user’s device.
Once several devices have started sharing, other devices request the pieces from each peer, reassembling the software and installing it locally. If the device is configured as a peer, it will cache parts of the software for others to retrieve.
Autopatching
Using lessons learned from WUfB, Microsoft has started a new managed update service called AutoPatch. While still based on the Update Rings from WUfB, Microsoft provides all the heavy lifting for configuring, testing, and deploying updates instead of an organization’s patch management team
Think of AutoPatch as managed services lite provided through Microsoft. An organization opts into the AutoPatch program and Microsoft ingests their devices into an automated patching system that tests, pilots, and fully deploys patches. If Microsoft finds a patch that is problematic, they will place a hold on the patch until the problem is cleared up, at which time they will release the patch to install.
AutoPatch is intended to be used by small to mid-size organizations that have limited staff and are not constrained by specialized software. When a company adopts AutoPatch, they can be assured that their systems will be as up-to-date as possible.
Can I Automate All of My Patching?
If you want to see a fight break out at an IT conference, ask a room full of people whether or not to automate their patch management. While it might seem that automation can save time and manpower, some argue that automated patching can introduce just as many problems as they solve.
We’ll be the first to admit that some organizations have been burned. They jumped in with an automated solution that was not set up correctly in accordance with a patch management strategy, so there were problems. Sometimes severe problems.
Here’s the thing. Patch management itself does not vary much from industry to industry, or environment to environment. Once you know the best practices and have refined the process, it can be automated and deployed easily.
But there’s the trick. Using a “refined process.”
Consider server order. If one deployed an automated solution to patch a multi-tiered server structure without taking into account boot order, it will lead to problems. And it might take an IT department hours or days to figure out what went wrong.
Another example: deploying a patch to an entire workgroup without thinking through exceptions. We once worked with a law firm that used a legacy plug-in for document management, a critical application in their business. A new patch was deployed that was incompatible with the plugin and things ground to a halt. Again, automation wasn’t so much the problem as was the failure to think through the various use cases.
However, if the needed business rules are well defined and ready to put into place, an automated patch management solution can be deployed quickly. In as little as an hour. From then on, patch deployments take no more than a week. That’s a time-saving most organizations can get behind.
Even better, automating the process means that IT staff no longer need to manually babysit the patch management process. They’re free to do more strategic and profitable things. Business continuity can be maximized even while vulnerabilities are addressed.
How to Automate
Outside of Windows Update for Business and Autopatch, most organizations use a third party maintenance tool to help manage and automate patching – especially when it comes to servers. Many of these integrate well with both WSUS and MCM/SCCM.
To be honest, buying such tools for server update management is often unnecessary. Microsoft’s own tools are up to the job, but they need to be expertly configured. Issues with automation don’t go away by throwing more automation software at the problems. They go away by following best practices.
Patching servers in a data center is a little different from patching your typical workstation or endpoint so a slightly different approach is needed. Not only are uptime and server security much bigger concerns, but there are different approaches that one can take.
Step 1 | Identify Device Connectivity Scenarios and Point of Authority
“Servers” is almost a misleading term, because there are so many different scenarios these days. Many do not look like the classical server that an IT department would keep in a closet somewhere on prem (though that does count, too). IT professionals talk about data centers, but that also seems restrictive. Therefore, it helps to think through the different server connectivity scenarios:
In-office/on-prem devices – in other words, hardware that exists on site. There are many ways to update and patch these servers: Group Policy Objects (GPOs) through Windows Server Update Services (WSUS), and MCM/SCCM to name a few.
Servers in the DMZ – DMZs are a sort of “buffer zone” between the public internet and your internal network(s). A good example would be web servers that have some public-facing components. Patching servers in a DMZ often requires careful management of firewall rules and/or placing an update server in the DMZ.
Standalone cloud devices – Cloud servers are sometimes standalone and have little-to-no network connectivity to production management systems. That being the case, using SaaS (software as a service) tools such as Microsoft’s Intune or OMS would be the way to go for updates. Given that they have support for multiplatform and are network agnostic, these tools would be preferred point of authorities.
Step 2 | Identify Production Deployment Strategy
Patching servers effectively will require a standard deployment process according to known business rules. This deployment process follows the test, pilot, deployment, and assessment stages similar to the Update Rings found in Windows Update for Business, only performed through another tool.
Also be sure that your production deployment strategy works in the necessary feedback loops. Verifying success early on—or catching problems—depends on it.
Step 3 | Determine Server Patching Groups
Patching groups can be defined in many ways. Some of the more common ones include:
- Server owner
- Service (application hosting, communications, web hosting, file serving, and so on)
- Criticality
- Redundancy with other devices
Servers will need to be added to groups once you have your deployment strategy settled. Be sure that each server is in a group!
When dealing with multiple servers, first identify any dependencies that might require a certain server reboot order. Otherwise you might find that a given server can’t access needed data even though the patch works just fine. For example, it is best practice to bring down a multi-tier system by starting with the presentation tier (web server), then the application tier, and then finally the database tier. These systems should then be brought back up in reverse order.
The Easy Button for Modern Microsoft Patch Management
What’s the lesson here? Good patching is essential to a secure and efficient organization. Microsoft provides quality tools and features to do the job, especially with their modernized patching process. However, using refined processes are essential to implementing automation without problems or gaps in your security.
In our experience, only the largest organizations have the resources and expertise to adeptly deploy these tools and processes. Many companies can benefit from third-party software maintenance, along with modernizing their Microsoft patching to keep endpoints compliant and secure. Our M365 Endpoint Solutions are perfect for these organizations.
Model Technology Solutions
Would you like extra guidance or manpower to tackle endpoint projects? Would you like some extra help managing patching in your environment? If so, you can learn more here, and see if our Unified Endpoint Management Services are a good fit for your organization.
We’re a dedicated Microsoft Solutions Partner with 60,000 endpoints currently under our management. We can partner with your internal team or work on our own to complete projects and help you modernize your patching environment. We help our clients to get their endpoints to an average of 96% compliance through our managed services (that includes servers and mobile devices). Our clients also see boosts to their M365 security score, and they spend less time and resources managing their ongoing patching processes.
Future Posts
In upcoming posts in this series, we will go deeper into configuring and managing patches and operating systems. We will discuss best practices, configuring update rings, third-party patching, and give you pointers on how to manage your device update using these modern tools.
See you in future posts!
Additional resources:
Windows Update for Business – Windows Deployment | Microsoft Learn
How Microsoft is transforming its own patch management with Azure | Inside Track Blog
Azure Automation Update Management overview | Microsoft Learn
