Metrics endpoint is giving 500 error

[necipakca]

After upgrading the operator to v0.30.0, we lost the Prometheus metrics.
I investigated a bit and found a problematic VulnerabilityReport.
I’m attaching the report to the issue, but I’m not sure which part of it is causing the error.
Also, I can’t see any logs related to the metrics endpoint.

vul-metrics-corrupted.yml

In Pod :

/ $ wget -O - localhost:8080/metrics
Connecting to localhost:8080 ([::1]:8080)
wget: server returned error: HTTP/1.1 500 Internal Server Error

I think the problem is there is magical three dot under the title field.

  - fixedVersion: 2.56.1-9.amzn2.0.8
    installedVersion: 2.56.1-9.amzn2.0.6
    lastModifiedDate: "2024-11-21T05:27:19Z"
    links: []
    packagePURL: pkg:rpm/amazon/glib2@2.56.1-9.amzn2.0.6?arch=x86_64&distro=amazon-2+%28Karoo%29
    primaryLink: https://avd.aquasec.com/nvd/cve-2020-35457
    publishedDate: "2020-12-14T23:15:12Z"
    resource: glib2
    score: 7.8
    severity: HIGH
    target: ""
    title: GNOME GLib before 2.65.3 has an integer overflow, that might lead to a
      ...
    vulnerabilityID: CVE-2020-35457

[peterD4N]

We are experiencing the Same Issue on v0.27.3 (chart v0.29.3) with operator.metricsVulnIdEnabled: true

The issue seems like is caused by unescaped double-quote characters " in vulnerability title fields that are used as Prometheus label values (vuln_title).
Prometheus's text exposition format requires \, ", and \n to be escaped inside label values, but the operator emits them raw.

Behaviour:

• With two operator replicas, typically only one pod returns HTTP 500 while the other returns HTTP 200 — likely a race condition where the first replica to collect/cache the corrupted metrics becomes the broken one.
• The healthy pod has zero error-level logs; the broken pod may also have zero errors — corruption happens silently inside the Prometheus client library when building the text output.

[afdesk]

Hi guys!
@necipakca @peterD4N thanks for the report! and thanks for the details.
it's really really useful!

it’s very unlikely the 500 is caused by unescaped quotes in label values. The Prometheus Go client used here escapes label values for text exposition automatically.

Likely root cause - duplicate label names in a metric descriptor.

I'm on it right now and try to clarify it.

[afdesk]

but I'll check unescaped quotes too

[peterD4N]

@afdesk thanks for looking into this.

Update from us:
I got the actual 500 response body. It's not duplicate label names in the descriptor — it's duplicate time series (same metric name + same label values emitted twice in Collect()):

An error has occurred while serving metrics:

6 error(s) occurred:
collected metric "trivy_vulnerability_id" { label:{name:"class" value:"lang-pkgs"} label:{name:"container_name" value:"knowledge-base-service"} ... label:{name:"vuln_id" value:"CVE-2026-25990"} label:{name:"vuln_title" value:"pillow: Pillow: Out-of-bounds Write via Specially Crafted PSD Image"} gauge:{value:1}} was collected before with the same name and label values

Root cause: duplicate vulnerability entries within the same VulnerabilityReport CRD.

The Trivy scanner writes the same CVE multiple times within a single report (identical vulnerabilityID, resource, target, pkgPath, installedVersion, fixedVersion, severity, score, title, etc.). When collectVulnerabilityIdReports() iterates over report.Vulnerabilities, it emits MustNewConstMetric with identical label values twice, and Go's Prometheus client rejects it.

Example — report replicaset-554b7986d9 has CVE-2026-25990/pillow listed 2 times with identical fields:

Entry 0: target=Python pkgPath= installedVersion=12.1.0 fixedVersion=12.1.1
Entry 1: target=Python pkgPath= installedVersion=12.1.0 fixedVersion=12.1.1

With 2 replicas, only the leader pod registers the custom Prometheus collector — the non-leader serves basic Go metrics (200). So it's always one pod 500, one pod 200.

The fix should probably be deduplication in collectVulnerabilityIdReports() before calling MustNewConstMetric, or deduplication in the scanner when writing the report. A simple seen map on the label tuple would prevent the 500.

Looks like this PR will help.

[afdesk]

@peterD4N thanks for the details.
you're right - #2884 should fix it.

[dkulchinsky]

Thanks for the fix @afdesk, is there an ETA for when 0.31.0 will be released?