ServiceNow Discovery: MID Server Design
Introduction
MID Servers are the engine of your ServiceNow Discovery and the extension of the ServiceNow Platform in your environment.
Before getting to the design, however, make sure you have aligned on the correct MID Server Architecture.
As I started working on ServiceNow Discovery, I faced several issues, which at first, I was not directly able to correlate to the deficiencies of the MID Server Design.
Many of you have faced or will face the following scenarios:
At one customer, we did thorough discovery troubleshooting in non-Prod, only to realize that once everything was migrated to Prod, only a small percentage of the CIs discovered in non-Prod were being discovered in Prod. This happened even though the Credentials and Discovery Schedules were accurately migrated to Prod.
For another project, while ServiceNow Discovery did run as scheduled, the variance between each discovery run was very high. Sometimes it ran in a few minutes, sometimes it took a few hours. This resulted in exceeding the allocated time-window for discovery.
The worst, however, was during another deployment, where some Discovery Schedules would randomly hang thereby preventing Discovery to complete. This seriously undermined the credibility of ServiceNow Discovery and the team responsible for deploying it. Yikes!
At these times, it’s easy to blame these issues on ‘ServiceNow’. Which leads to us wishing for the following ‘simple’ requirement…
My only requirement is that ServiceNow Discovery runs consistently, completes successfully and takes the same amount of time to run, every time it runs.
On the brighter side though, there were many projects where we were able to meet this rather ‘simple’ requirement.
So, for starters I concluded that it was not ‘ServiceNow’. Then, I looked deeper into the nuances of these implementations and realized the following commonalities in the MID Server Design, that really made all the difference.
As a result, we recommend the following to every customer we work with. And we have seen ServiceNow Discovery run consistently, complete successfully and take the same amount of time to run, every time it runs. Win! Win! Win!
MID Server Design
It’s important to incorporate all three recommendations below to get the consistency for ServiceNow Discovery across all your environments.
Don’t forget to configure MID Server Properties to further fine-tune Discovery.
One MID Server per Host per ServiceNow Instance
I can’t emphasize the importance of having a single MID Server on a single host per ServiceNow Instance. Its such a simple thing to enforce, yet, too may implementations, (including some of my earlier deployments) are guilty of not following this approach.
This results in predictable availability of the MID Server Host resulting in predictable performance of the MID Server, which is more important than ever, given more and more processing is now being pushed to the MID Server.
This is what you should not have running on the same Host:
Other MID Servers for Discovery pointing to the same ServiceNow Instance (I repeat)
Other MID Servers for other things (like Integration, Orchestration, etc) pointing to the same ServiceNow Instance
Other significant Apps, like monitoring, scanning, etc, that may result in resource contention
Barring the three exceptions above, there IS however, one valid and recommended use-case for having another MID Server on the same host, which is described in the next section…
Non-Prod MID Server should be on the same host as Prod MID Server
We often come across a question of whether non-Prod MID Servers should be installed on the same host the Prod MID Server is running on, or a different host.
While it may be natural to request different hosts for Prod vs non-Prod MID Servers, the non-intuitive recommendation is to have both the Prod and non-Prod MID Servers run on the same host!
This often raises immediate concerns of the ‘violation’ of integrity where Prod and non-Prod resources should not share the same infrastructure.
Here is the rationalization to address these concerns:
The MID Servers are completely independent of each other from an App footprint and execution standpoint.
It’s obvious not to have Prod and non-Prod Discovery run at the same time. So this eliminates the resource contention at the onset. On the other hard, if there are issues with Prod Discovery, having another MID Sever running on the same host allows for the best scenario for doing a targeted discovery for Troubleshooting, allowing for quicker resolution to Prod issues.
Network Firewall Rules and ACLs on SNMP Devices are important configurations that need to take place outside of ServiceNow for Discovery to work consistently. Bring on the same host, when these configurations are tested in non-Prod, we can be sure they will work in Prod. Ironically, there isn’t any other way to ensure this apart from having the non-Prod and Prod MID Servers running on the same Host.
In short, this design provides a solid grounding to ensure that discovery validated in non-Prod works the same way in Prod.
Co-locate Clustered MID Servers on the same Subnet
The only thing worse than something not working, is something not working some of the time. Think auto-pilot, elevators, alarm clocks… you get the picture.
And ServiceNow Discovery falls in the same category. Try troubleshooting an intermittent issue. We have all been there…
For starters, here is why you should always use MID Server Clusters.
While clusters provide the scale we need to discover a large number of CIs, its obvious that each node in the cluster must behave exactly the same way, failing which we fall right into the sweet spot we all hate to be in, which is ‘intermittent issues’.
Based on the recommendations above, it doesn’t make sense to have two Clustered MID Servers running on the same host, especially, when the MID Server Properties are configured to optimally use the MID Server host.
Therefore, with Clustered MID Server hosts, co-locating these hosts on the same subnet is key.
Conclusion
Its obvious that 5 mid servers on one host is slower than 5 mid servers on 5 mid Server hosts. More importantly though, with the additional hardware, you may only then need 2 MID Servers vs 5 and the discovery runs will be more predictable.
So here is a simple formula:
Number of MID Server Hosts = Number of MID Servers in Prod
Customers who have followed the recommendations above experience flawless execution of ServiceNow Discovery.
