Cloud Security Compare

Buyer note

How to run a CNAPP proof of concept between two shortlisted vendors

Cloud Security Compare editors · · Editorial assessment

The short version

A CNAPP proof of concept is a test of two products against the same accounts, the same questions and the same scoring sheet. Decide the criteria and weights before either vendor connects, run both on the same slice of your estate at the same time, and score what you saw rather than what the demo showed. This note turns our seven comparison criteria into a test plan.

What should you decide before either vendor connects?

Most proofs of concept go wrong before they start. Each vendor gets a different set of accounts, a different contact and a different idea of success, and the result reflects the setup more than the product. Fix three things first.

  • Scope. Give both vendors the same cloud accounts, subscriptions or projects. Include at least one account with production-like workloads, a Kubernetes cluster if you run Kubernetes, and the AI services you already use.
  • Criteria and weights. Write them down and send them to both vendors. Our seven criteria are a starting point: agentless coverage 18%, runtime protection 16%, risk prioritization 18%, code-to-cloud 14%, AI workloads 12%, ecosystem 12% and pricing transparency 10%. Change the weights to fit your estate before you see results, not after.
  • Owners. Name one person on your team for each track (posture, runtime, code, AI) who runs the tests and signs off the score for that track.

The rubric behind our weights is on How to read these comparisons.

How do you test agentless coverage fairly?

Connect both products to the same accounts on the same day and compare the inventory each one returns. Vendors connect in different ways. Wiz connects by cloud API. Orca Security's SideScanning reads workloads' block storage and rebuilds the file system in a read-only view, and Orca says it deploys in minutes. Microsoft Defender for Cloud includes agentless VM and Kubernetes scanning only in its paid Defender CSPM and Defender for Servers Plan 2 plans, so check which plan the trial turns on.

  • Which resource types appear in one inventory and not the other: VMs, container images, serverless functions, managed databases, AI services?
  • Which clouds are covered? Check every cloud you run, not only the main one. Orca names six clouds, including Oracle Cloud, Alibaba Cloud and Tencent Cloud; Wiz names AWS, Azure, Google Cloud and OCI.
  • How long did it take from connection to a complete inventory? Time it yourself rather than using the vendor's figure.
  • What permissions did each product ask for, and did your cloud team approve them unchanged?

How do you test runtime protection without risking production?

Runtime sensors run inside your workloads, so start on a non-production cluster or a small group of hosts. Most sensors in this category are built on eBPF: the Wiz Sensor, the Orca Sensor and Upwind's sensors all are, and Sysdig's runtime detection is built on Falco. Ask each vendor what the sensor does beyond alerting. Orca says its sensor can be configured to terminate processes. Wiz says its sensor blocks threats. Aqua Security says its kernel-layer enforcement blocks attacks without killing the container.

  • Install time, and the change process it needed (Helm chart, DaemonSet or host package).
  • CPU and memory use on your own workloads, measured with your own monitoring.
  • Which detections fired for a planned, harmless test, and how long each took to appear.
  • Whether blocking is on by default, and who in your team can turn it on.

How do you compare risk prioritization?

Every vendor in our matrix says it ranks findings by exploitability rather than severity alone. The fair test is to give both products the same estate and compare the top of each list. Export the ten highest-priority findings from each product and ask three questions of every item: is it real, is it reachable, and would we fix it this week? Count the yes answers. A top ten your team would act on means the ranking works. A top ten that needs a second round of triage means it does not.

Then read how each product explains a finding. Wiz presents attack paths through its Security Graph. Orca runs attack path analysis on one data model built from agentless context. CrowdStrike says its Cloud Risk Engine, announced in March 2026, maps cloud risks to adversary tradecraft. SentinelOne shows Verified Exploit Paths. The explanation matters as much as the rank, because it is what your engineers will read before they fix anything.

How do you test code-to-cloud and AI coverage?

Connect the same repositories to both products. Check which source control and CI systems each one supports: Orca lists GitHub, GitLab and Azure DevOps, plus Jenkins, Bitbucket and CircleCI; Wiz Code lists GitHub and IDE scanning. Then pick one cloud finding and ask each product to trace it back to the file and the owner that caused it.

For AI, list the AI services and models you run and check which ones appear in each product's AI inventory. If your teams use MCP servers or AI coding tools, ask whether the product can see them. Orca and Upwind both name MCP servers on their AI security pages. Our note on AI workload security sets out what each vendor lists.

How do you compare cost when neither vendor publishes a price?

Of the nine platforms we compare, only Microsoft Defender for Cloud publishes per-resource pricing with a cost calculator. For the others, ask both vendors the same three questions in writing: what is the billable unit, how is it counted in your estate, and what happens to the price when the estate grows. Sysdig publishes its unit (number of hosts) but not the price. Defender for Cloud counts some resources in groups; eight AWS Lambda functions make one billable unit. Put both quotes on the same unit before comparing them.

How should you score the result?

Use the sheet you wrote before the test. Score each criterion 0 to 5 for each vendor with a one-line reason, multiply by your weights and add up. Keep the reasons: you will need them when someone asks why you chose the product that scored lower on one criterion. If the two totals are within a few tenths of each other, look at the criteria where the gap is widest and decide on those. That is how our head-to-head pages are built, and the tug-of-war bars on Wiz vs Orca or Orca vs Upwind show where the widest gaps sit.

Sources

Related pages