Add a per-table SHADOW enforcement mode for storage and query quotas #19837

Problem

Pinot currently enforces storage and QPS quotas as soon as they are configured on a table. This makes it risky to introduce or adjust quotas for existing tables.

• Storage quota: Segment uploads are rejected with HTTP 403, and realtime consumption is paused when the quota is exceeded.
• QPS quota: Queries exceeding the configured limit are rejected with HTTP 429.

There is no way to see how many queries or uploads would have been rejected by a proposed quota without actually enforcing it. We need a way to evaluate quota limits before enabling enforcement.

Proposal

Introduce a QuotaEnforcementMode enum with three modes: ENFORCED, SHADOW, and DISABLED.

Add two optional fields to QuotaConfig to control storage and query quota enforcement independently:

{
  "quota": {
    "storage": "10G",
    "maxQueriesPerSecond": "100",
    "storageQuotaEnforcementMode": "SHADOW",
    "queryQuotaEnforcementMode": "SHADOW"
  }
}

Expected Behavior

• Storage quota: In SHADOW mode, segment uploads should not be rejected and realtime consumption should not be paused due to quota breaches. Tables already paused due to storage quota should resume when switched to SHADOW.
• Query quota: In SHADOW mode, queries exceeding the configured QPS quota should still be served.
• Metrics: SHADOW and ENFORCED should emit the same quota breach metrics (BrokerMeter.QUERY_QUOTA_EXCEEDED and ControllerMeter.STORAGE_QUOTA_EXCEEDED), allowing operators to evaluate limits before enforcing them.
• Backward compatibility: Tables without the new fields should continue using ENFORCED, preserving existing behavior.