OT Security Platform Evaluation: A Practitioner's Guide

Before you talk to vendors

The most common evaluation mistake is starting with vendor conversations. Before any outreach, you need an accurate picture of your own environment. Vendors will ask about it. More importantly, your answers determine which platforms are actually worth evaluating.

Know your asset inventory. Most organizations discover during evaluation that their asset inventory is incomplete or stale. A visibility platform that finds 40% more assets than you knew existed is not a selling point; it is a data point about your current blind spots. Know this going in.

Audit your protocol stack. List every industrial protocol running in your environment. Include legacy protocols on aging equipment. This list determines which platforms can cover your environment and which cannot, regardless of what their marketing materials claim. Vendors with strong Modbus and EtherNet/IP coverage may have thin coverage for IEC 61850 or GOOSE if you operate grid infrastructure. The protocol audit is non-negotiable.

Document your deployment constraints. Active scanning is not acceptable in most OT environments. Air-gapped segments cannot phone home to a cloud platform. Legacy assets cannot run agents. Capture these constraints explicitly — they will eliminate platforms from consideration faster than any feature comparison.

Identify your compliance obligations. If you are under NERC CIP, NIS2, TSA directives, or IEC 62443, list the specific requirements that drive this procurement. Compliance requirements often determine vendor category before anything else does. A NERC CIP-015 obligation points directly to passive OT network monitoring. A NIS2 24-hour incident reporting requirement points to detection and logging capabilities with automated notification workflows.

Shortlisting: from the full market to three to five platforms

Do not evaluate every vendor in the market. The evaluation process is disruptive enough without running six parallel PoCs. Get to three to five platforms before any formal engagement.

Start with functional category fit. The four vendor categories — OT visibility and detection, IT/OT converged platforms, OT enforcement and protection, and specialists — solve different problems. If you do not have OT visibility today, a converged platform that requires existing OT data to be meaningful is the wrong starting point. Match the category to the gap you are closing first.

Make the tier decision. Established platforms bring integration depth, enterprise support, and proven deployments in environments like yours. Challengers bring tighter focus and often better performance on specific problems. The decision depends on your organization's risk tolerance for unproven vendors, your internal OT security maturity, and whether you need the support infrastructure that established vendors provide. There is no universally correct answer, but the decision should be explicit, not defaulted.

Apply deployment model as a filter. A cloud-only platform is disqualifying if you have air-gapped segments that cannot reach it. An agent-based platform is disqualifying if your legacy assets cannot support agents. Apply these filters before reaching out to any vendor.

Apply compliance alignment as a filter. If NERC CIP compliance reporting automation matters to your procurement, platforms without it require manual evidence collection — a meaningful operational cost that should be factored in.

The RFP and vendor briefing

Send a written briefing document to shortlisted vendors before any demo. This serves two purposes: it tells vendors what matters to you, and it tests whether they actually read it. A vendor who demos capabilities you explicitly said you do not need has already told you something about how they will approach implementation.

The briefing document should include your protocol list, your deployment constraints, your compliance obligations, and your top three evaluation criteria. Ask vendors to confirm in writing how their platform addresses each one before scheduling a demo.

Ask about protocol coverage specifically. "Do you support Modbus?" is the wrong question. Ask for the specific version and function codes supported, and ask how coverage was validated, as passive deep-packet inspection and active querying produce different coverage claims. Ask specifically about the protocols on your list that are not Modbus or EtherNet/IP.

Test passive-only claims. Ask how asset discovery works in a passive-only deployment. Ask whether any features require active queries. Ask specifically what happens if an active query reaches a legacy PLC that was not designed to handle unexpected network traffic. Vendors with genuine passive-only architectures answer this without hesitation. Vendors whose "passive" deployment has exceptions will hedge.

Ask about professional services requirements. Some platforms require vendor professional services for deployment, tuning, and ongoing operation. This is not necessarily disqualifying — but it needs to be in your TCO calculation and your internal resource plan from the start, not discovered after purchase.

Ask about your specific compliance obligations. Ask the vendor to walk through exactly how their platform satisfies your specific CIP standards or NIS2 requirements. Ask what evidence the platform generates and in what format. Ask whether they have customers who have been audited under these frameworks and what the auditor's experience with their evidence package was.

What good and evasive answers look like

The proof of concept

A PoC should test your environment, not a vendor's demo environment. If a vendor will only run a PoC in their lab against their test assets, that tells you something. Push for a limited deployment in a representative segment of your actual network.

Define success criteria before the PoC starts. Write them down and share them with the vendor. Success criteria should include asset discovery completeness (compare against your known inventory), false positive rate on normal operations, protocol coverage against your specific equipment, and operational impact during deployment. If the vendor objects to written success criteria, that is a red flag.

Test against your most challenging assets. Every platform performs well against modern, well-documented equipment. Test against the oldest, most proprietary assets in your environment: the 15-year-old PLC running a custom protocol, the historian on Windows Server 2008. This is where platforms differentiate.

Measure false positive rate against real operations. Run the platform during a normal engineering maintenance window. Count how many alerts it generates on known-good activity. A platform that cannot be tuned to distinguish between a cyberattack and a planned maintenance session will generate alert fatigue that causes your SOC to ignore it.

Evaluate the interface for the people who will actually use it. If the primary user is an OT engineer, the interface needs to make sense to someone who thinks in terms of PLCs and process flows, not IP addresses and CVEs. If the primary user is a SOC analyst, it needs to map to the workflows they already use. Run the interface past the actual end users during the PoC, not just the security team.

Getting plant manager buy-in

A platform that the CISO approves and the plant manager blocks does not get deployed. The plant manager's concerns are legitimate operational risk concerns, not obstruction. Address them directly rather than working around them.

The core concern is that the security tool itself becomes an operational risk — that it will generate traffic that affects a sensitive process, that a misconfiguration will cause a false isolation event, that deployment will require downtime. These are not hypothetical concerns. They have happened.

Address them with specifics. Show the plant manager exactly how the platform deploys: which network ports it connects to, what traffic it generates (if any), what actions it can and cannot take autonomously. If the platform is passive-only with no autonomous response capability, state that clearly and document it. If the platform has enforcement capabilities, document exactly what conditions trigger them and confirm they can be disabled or restricted during initial deployment.

The plant manager also needs to understand the operational risk of not deploying. The OT Security Regulatory Map and threat data from the Why page address the same risk calculus the plant manager applies to every other operational decision.

Making the decision

When two platforms are close, the tiebreakers in rough order of importance are: protocol coverage depth on your most challenging assets, compliance evidence quality for your specific obligations, support model fit for your internal team's maturity, and deployment track record in environments like yours.