Skip to content

Commit 508a30d

Browse files
alongubkinisloclaude
authored
feat: check GCP compute and project IAM quotas (#1028)
GCP deployments now check regional/family CPU and supported GPU capacity, project custom roles including soft-deleted roles, and service accounts against applied limits. Demand comes from resource configuration, separates immediate launches from maximum headroom, and credits only proven held identities. Desired role IDs matching this deployment’s own soft-deleted roles are undeleted and do not consume a new role slot. Explicit custom-role/service-account capacity failures map to the same typed project quota guidance; ordinary throttling retains provider retry behavior. Project limits use Cloud Quotas and Service Usage applied values; regional Compute responses supply provider usage. IAM metadata lists paginate completely. Service accounts include disabled/default accounts; partial, duplicate or missing identities yield Unknown. Cloud Quotas does not supply all-regions usage on the observed target, so readable all-regions limits without usage remain Unknown. Unresolved dimensioned GPU variants also remain Unknown. Setup enables `cloudquotas.googleapis.com` through existing service activation, including Terraform setup, and both test-infrastructure modules declare it. Older setup imports and installed stacks without that optional activation keep it absent; disabled or denied reads warn with an enable/API-permission hint rather than blocking updates. Permissions: `compute.regions.get`, `compute.projects.get` and `cloudquotas.quotas.get` in project-scoped provisioning read quota/capacity metadata. Service-account provision/management/heartbeat sets grant project `iam.serviceAccounts.list` for complete identity inventory; resource-scoped viewer grants cannot list project inventory. `serviceusage.quotas.get` reads project applied IAM limits through a narrow shared metadata role; that role is included in demand. Existing IAM custom-role metadata reads are reused. No token/key/customer-content or broad telemetry access is added. Existing public-safe implementation/read evidence and exact limitations are retained in [#1010](#1010): applied CPU/GPU all-regions limits360/360; A2 CPU12/A100 GPU1 controls; paginated IAM identity equivalence and two-probe create/delete counting; custom-role capacity response replay. These are existing integrated-feature controls, not new runs on the split commit. Do not interpret a parent accepting a min0/max2 configuration as a late GPU quota failure; no GPU VM was launched. No new parent/race/entry-point claim is added by this split. [Regional CPU negative](https://raw.githubusercontent.com/alienplatform/alien/pr-assets/alien-5/gcp-cpu-reader/negative.png), [positive reader control](https://raw.githubusercontent.com/alienplatform/alien/pr-assets/alien-5/gcp-cpu-reader/control.png), and [custom-role config controls](https://raw.githubusercontent.com/alienplatform/alien/pr-assets/alien-5/custom-roles-config-controls.png) show the existing read/config evidence. All these controls left provider inventories unchanged and created no resources. Compatibility: no persisted or SDK schema fields change in this layer. Missing management-role quota reads produce non-blocking Unknown. Old GCP handoff import and subsequent mutation regression coverage retain ownership validation. The supported generator and generated-output diff both passed in [feature CI38534727697](https://github.com/alienplatform/alien/actions/runs/38534727697); current-head CI and review also pass. Compute blocks only when capacity certainly launched by this operation does not fit. Initial launch uses the configured minimum; scale-to-maximum shortfalls warn visibly with the quota identity, shortfall and increase link. Bounds-only changes add no replacement surge. Instance-type replacement, and effective disk changes on AWS, count the proven rolling overlap. Late-resolved image/identity/interrupted-rollout inputs remain a visible Unknown warning; they never add speculative blocking surge. Observed held capacity is credited in its original quota pool. Layer validation: 518 preflight library tests pass. Read-only policy control: regional CPUS4/200 and N2_CPUS0/200. No new cloud resources or quota overrides were needed. Merge order: #1032 → #1026 → #1027 → #1028 → #1029 → #1030. Each layer depends on the preceding PR; human merges only. Service Usage metric requests use `view=FULL` to receive applied quota buckets/effective limits. The missing-query parent regression fails; all 5 relevant wire tests pass, including denied reads. Current head `1d7ab07ce`: **CI green, Greptile 5/5, mergeable and ready for review** ([review](#1028 (comment))). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: islo <sandbox@islo.dev> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent d4ac8fd commit 508a30d

37 files changed

Lines changed: 3871 additions & 18 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
//! Read applied quota values from the Cloud Quotas API.
2+
3+
use crate::gcp::{
4+
api_client::{GcpClientBase, GcpServiceConfig},
5+
GcpClientConfig,
6+
};
7+
use alien_client_core::Result;
8+
use reqwest::{Client, Method};
9+
use serde::{Deserialize, Serialize};
10+
use std::fmt::Debug;
11+
12+
#[cfg(feature = "test-utils")]
13+
use mockall::automock;
14+
15+
#[derive(Debug)]
16+
pub struct CloudQuotasServiceConfig;
17+
18+
impl GcpServiceConfig for CloudQuotasServiceConfig {
19+
fn base_url(&self) -> &'static str {
20+
"https://cloudquotas.googleapis.com/v1"
21+
}
22+
fn default_audience(&self) -> &'static str {
23+
"https://cloudquotas.googleapis.com/"
24+
}
25+
fn service_name(&self) -> &'static str {
26+
"Cloud Quotas"
27+
}
28+
fn service_key(&self) -> &'static str {
29+
"cloudquotas"
30+
}
31+
}
32+
33+
#[cfg_attr(feature = "test-utils", automock)]
34+
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
35+
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
36+
pub trait CloudQuotasApi: Send + Sync + Debug {
37+
/// Get an applied quota, including its dimension-specific values.
38+
async fn get_quota_info(&self, service: &str, quota_id: &str) -> Result<QuotaInfo>;
39+
}
40+
41+
/// Cloud Quotas client restricted to reads in the configured target project.
42+
#[derive(Debug)]
43+
pub struct CloudQuotasClient {
44+
base: GcpClientBase,
45+
project_id: String,
46+
}
47+
48+
impl CloudQuotasClient {
49+
pub fn new(client: Client, config: GcpClientConfig) -> Self {
50+
Self {
51+
project_id: config.project_id.clone(),
52+
base: GcpClientBase::new(client, config, Box::new(CloudQuotasServiceConfig)),
53+
}
54+
}
55+
}
56+
57+
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
58+
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
59+
impl CloudQuotasApi for CloudQuotasClient {
60+
/// https://cloud.google.com/docs/quotas/reference/rest/v1/projects.locations.services.quotaInfos/get
61+
async fn get_quota_info(&self, service: &str, quota_id: &str) -> Result<QuotaInfo> {
62+
let path = format!(
63+
"projects/{}/locations/global/services/{}/quotaInfos/{}",
64+
self.project_id,
65+
urlencoding::encode(service),
66+
urlencoding::encode(quota_id),
67+
);
68+
self.base
69+
.execute_request(Method::GET, &path, None, Option::<()>::None, quota_id)
70+
.await
71+
}
72+
}
73+
74+
/// Applied quota metadata. This API does not report current usage.
75+
#[derive(Debug, Clone, Serialize, Deserialize)]
76+
#[serde(rename_all = "camelCase")]
77+
pub struct QuotaInfo {
78+
/// Provider quota identifier.
79+
pub quota_id: String,
80+
/// Provider metric identifier.
81+
pub metric: String,
82+
/// Dimension keys; empty for project-wide quotas.
83+
#[serde(default)]
84+
pub dimensions: Vec<String>,
85+
/// Applied values for dimension groups.
86+
pub dimensions_infos: Option<Vec<DimensionsInfo>>,
87+
}
88+
89+
/// Applied value for one dimension group.
90+
#[derive(Debug, Clone, Serialize, Deserialize)]
91+
#[serde(rename_all = "camelCase")]
92+
pub struct DimensionsInfo {
93+
/// Applied quota, distinct from the catalog default.
94+
pub details: Option<QuotaDetails>,
95+
}
96+
97+
/// Applied int64 quota value encoded as a decimal string.
98+
#[derive(Debug, Clone, Serialize, Deserialize)]
99+
#[serde(rename_all = "camelCase")]
100+
pub struct QuotaDetails {
101+
/// Missing or invalid values must not be treated as zero.
102+
pub value: Option<String>,
103+
}
104+
105+
impl QuotaInfo {
106+
/// Only a single dimensionless project-wide value is safe to use here.
107+
pub fn project_limit(&self) -> Option<f64> {
108+
if !self.dimensions.is_empty() {
109+
return None;
110+
}
111+
let [info] = self.dimensions_infos.as_deref()? else {
112+
return None;
113+
};
114+
let value = info
115+
.details
116+
.as_ref()?
117+
.value
118+
.as_deref()?
119+
.parse::<u64>()
120+
.ok()?;
121+
Some(value as f64)
122+
}
123+
}

‎crates/alien-gcp-clients/src/gcp/compute.rs‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,12 @@ impl GcpServiceConfig for ComputeServiceConfig {
7070
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
7171
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
7272
pub trait ComputeApi: Send + Sync + Debug {
73+
/// Reads applied regional quota limits and provider-counted usage.
74+
async fn get_region_quotas(&self, region: String) -> Result<QuotaCollection>;
75+
76+
/// Reads project quotas, including CPUs across all regions when present.
77+
async fn get_project_quotas(&self) -> Result<QuotaCollection>;
78+
7379
// --- Zone Operations ---
7480

7581
/// Lists zones in the project.
@@ -717,6 +723,20 @@ impl ComputeClient {
717723
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
718724
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
719725
impl ComputeApi for ComputeClient {
726+
async fn get_region_quotas(&self, region: String) -> Result<QuotaCollection> {
727+
let path = format!("projects/{}/regions/{region}", self.project_id);
728+
self.base
729+
.execute_request(Method::GET, &path, None, Option::<()>::None, "region")
730+
.await
731+
}
732+
733+
async fn get_project_quotas(&self) -> Result<QuotaCollection> {
734+
let path = format!("projects/{}", self.project_id);
735+
self.base
736+
.execute_request(Method::GET, &path, None, Option::<()>::None, "project")
737+
.await
738+
}
739+
720740
// --- Zone Operations ---
721741

722742
async fn list_zones(&self, filter: Option<String>) -> Result<ZoneList> {
@@ -6817,3 +6837,23 @@ mod tests {
68176837
);
68186838
}
68196839
}
6840+
6841+
/// Quota fields shared by Compute regions.get and projects.get.
6842+
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
6843+
#[serde(rename_all = "camelCase")]
6844+
pub struct QuotaCollection {
6845+
/// Absent quota data is distinct from zero usage.
6846+
pub quotas: Option<Vec<ComputeQuota>>,
6847+
}
6848+
6849+
/// Applied quota value and usage reported directly by Compute Engine.
6850+
#[derive(Debug, Clone, Serialize, Deserialize)]
6851+
#[serde(rename_all = "camelCase")]
6852+
pub struct ComputeQuota {
6853+
/// Provider metric identity, such as N2_CPUS.
6854+
pub metric: String,
6855+
/// Applied limit; absence must not become a catalog default.
6856+
pub limit: Option<f64>,
6857+
/// Current provider-counted usage.
6858+
pub usage: Option<f64>,
6859+
}

‎crates/alien-gcp-clients/src/gcp/iam.rs‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,13 @@ pub trait IamApi: Send + Sync + Debug {
6868

6969
async fn get_service_account(&self, service_account_name: String) -> Result<ServiceAccount>;
7070

71+
/// Read one page of account metadata in the configured target project.
72+
async fn list_service_accounts(
73+
&self,
74+
page_size: Option<i32>,
75+
page_token: Option<String>,
76+
) -> Result<ListServiceAccountsResponse>;
77+
7178
async fn patch_service_account(
7279
&self,
7380
service_account_name: String,
@@ -214,6 +221,32 @@ impl IamApi for IamClient {
214221
.await
215222
}
216223

224+
/// Lists service accounts without reading their IAM policies or credentials.
225+
/// https://cloud.google.com/iam/docs/reference/rest/v1/projects.serviceAccounts/list
226+
async fn list_service_accounts(
227+
&self,
228+
page_size: Option<i32>,
229+
page_token: Option<String>,
230+
) -> Result<ListServiceAccountsResponse> {
231+
let path = format!("projects/{}/serviceAccounts", self.project_id);
232+
let mut query_params = Vec::new();
233+
if let Some(size) = page_size {
234+
query_params.push(("pageSize", size.to_string()));
235+
}
236+
if let Some(token) = page_token {
237+
query_params.push(("pageToken", token));
238+
}
239+
self.base
240+
.execute_request(
241+
Method::GET,
242+
&path,
243+
Some(query_params).filter(|params| !params.is_empty()),
244+
Option::<()>::None,
245+
"serviceAccounts",
246+
)
247+
.await
248+
}
249+
217250
/// Updates (patches) a service account.
218251
/// See: https://cloud.google.com/iam/docs/reference/rest/v1/projects.serviceAccounts/patch
219252
async fn patch_service_account(
@@ -517,6 +550,17 @@ pub struct ServiceAccount {
517550
pub etag: Option<String>,
518551
}
519552

553+
/// One page of service-account metadata. Empty protobuf JSON lists can be absent.
554+
#[derive(Debug, Serialize, Deserialize, Clone)]
555+
#[serde(rename_all = "camelCase")]
556+
pub struct ListServiceAccountsResponse {
557+
/// Accounts returned on this page; callers must follow the continuation token.
558+
#[serde(default)]
559+
pub accounts: Vec<ServiceAccount>,
560+
/// Opaque cursor for the next page, absent on the last page.
561+
pub next_page_token: Option<String>,
562+
}
563+
520564
/// Request message for creating a service account.
521565
#[derive(Debug, Serialize, Deserialize, Clone, Default, Builder)]
522566
#[serde(rename_all = "snake_case")]

‎crates/alien-gcp-clients/src/gcp/mod.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ pub mod agent_platform;
22
pub mod api_client;
33
pub mod artifactregistry;
44
pub mod cloud_kms;
5+
pub mod cloud_quotas;
56
pub mod cloud_sql;
67
pub mod cloudasset;
78
pub mod cloudbuild;
@@ -20,6 +21,7 @@ pub mod pubsub;
2021
pub mod resource_manager;
2122
pub mod secret_manager;
2223
pub mod service_usage;
24+
pub mod service_usage_quotas;
2325

2426
use alien_client_core::{ErrorData, Result};
2527
use alien_error::{AlienError, Context, IntoAlienError};
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
//! Read applied quota values from the Service Usage quota API.
2+
3+
use crate::gcp::{
4+
api_client::{GcpClientBase, GcpServiceConfig},
5+
GcpClientConfig,
6+
};
7+
use alien_client_core::Result;
8+
use reqwest::{Client, Method};
9+
use serde::{Deserialize, Serialize};
10+
use std::collections::BTreeMap;
11+
use std::fmt::Debug;
12+
13+
#[cfg(feature = "test-utils")]
14+
use mockall::automock;
15+
16+
#[derive(Debug)]
17+
pub struct ServiceUsageQuotasServiceConfig;
18+
19+
impl GcpServiceConfig for ServiceUsageQuotasServiceConfig {
20+
fn base_url(&self) -> &'static str {
21+
"https://serviceusage.googleapis.com/v1beta1"
22+
}
23+
fn default_audience(&self) -> &'static str {
24+
"https://serviceusage.googleapis.com/"
25+
}
26+
fn service_name(&self) -> &'static str {
27+
"Service Usage quota"
28+
}
29+
fn service_key(&self) -> &'static str {
30+
"serviceusage"
31+
}
32+
}
33+
34+
#[cfg_attr(feature = "test-utils", automock)]
35+
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
36+
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
37+
pub trait ServiceUsageQuotasApi: Send + Sync + Debug {
38+
/// Get an applied quota, including its dimension-specific values.
39+
async fn get_quota_metric(&self, service: &str, metric: &str) -> Result<QuotaMetric>;
40+
}
41+
42+
/// Service Usage quota client restricted to reads in the configured target project.
43+
#[derive(Debug)]
44+
pub struct ServiceUsageQuotasClient {
45+
base: GcpClientBase,
46+
project_id: String,
47+
}
48+
49+
impl ServiceUsageQuotasClient {
50+
pub fn new(client: Client, config: GcpClientConfig) -> Self {
51+
Self {
52+
project_id: config.project_id.clone(),
53+
base: GcpClientBase::new(client, config, Box::new(ServiceUsageQuotasServiceConfig)),
54+
}
55+
}
56+
}
57+
58+
#[cfg_attr(target_arch = "wasm32", async_trait::async_trait(?Send))]
59+
#[cfg_attr(not(target_arch = "wasm32"), async_trait::async_trait)]
60+
impl ServiceUsageQuotasApi for ServiceUsageQuotasClient {
61+
/// https://cloud.google.com/service-usage/docs/reference/rest/v1beta1/services.consumerQuotaMetrics/get
62+
async fn get_quota_metric(&self, service: &str, metric: &str) -> Result<QuotaMetric> {
63+
let path = format!(
64+
"projects/{}/services/{}/consumerQuotaMetrics/{}",
65+
self.project_id,
66+
urlencoding::encode(service),
67+
urlencoding::encode(metric),
68+
);
69+
self.base
70+
.execute_request(
71+
Method::GET,
72+
&path,
73+
Some(vec![("view", "FULL".into())]),
74+
Option::<()>::None,
75+
metric,
76+
)
77+
.await
78+
}
79+
}
80+
81+
/// A quota metric and its applied limits from Service Usage.
82+
#[derive(Debug, Clone, Serialize, Deserialize)]
83+
#[serde(rename_all = "camelCase")]
84+
pub struct QuotaMetric {
85+
/// Fully qualified metric resource name.
86+
pub name: String,
87+
/// Provider metric identifier.
88+
pub metric: String,
89+
/// Applied limits; missing values are not a zero limit.
90+
pub consumer_quota_limits: Option<Vec<ConsumerQuotaLimit>>,
91+
}
92+
93+
/// A limit for one quota unit.
94+
#[derive(Debug, Clone, Serialize, Deserialize)]
95+
#[serde(rename_all = "camelCase")]
96+
pub struct ConsumerQuotaLimit {
97+
/// Metric whose limit this describes.
98+
pub metric: String,
99+
/// Provider unit, e.g. one per project.
100+
pub unit: String,
101+
/// Applied values grouped by dimensions.
102+
pub quota_buckets: Option<Vec<QuotaBucket>>,
103+
}
104+
105+
/// Applied quota for a dimension group. Catalog defaults are intentionally omitted.
106+
#[derive(Debug, Clone, Serialize, Deserialize)]
107+
#[serde(rename_all = "camelCase")]
108+
pub struct QuotaBucket {
109+
/// Dimension filters, empty for a project-wide quota.
110+
#[serde(default)]
111+
pub dimensions: BTreeMap<String, String>,
112+
/// Applied int64 value, encoded as a decimal string by the provider.
113+
pub effective_limit: Option<String>,
114+
}

‎crates/alien-gcp-clients/src/lib.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
pub mod gcp;
2+
pub use gcp::cloud_quotas::{CloudQuotasApi, CloudQuotasClient};
3+
pub use gcp::service_usage_quotas::{ServiceUsageQuotasApi, ServiceUsageQuotasClient};
24
pub use gcp::*;
35

46
// Re-export commonly used types for convenience

0 commit comments

Comments
 (0)