Skip to content

[Otel] fix otel spans and attributes #201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
majunze2001 merged 1 commit into from
Mar 4, 2026
Merged

[Otel] fix otel spans and attributes #201

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docker/dev.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,3 +55,6 @@ ENV OTEL_SDK_DISABLED=true
ENV VLLM_LOGGING_LEVEL=DEBUG

CMD ["bash"]

# export CORNSERVE_MOCK_SIDECAR=1
# OTEL_SDK_DISABLED=true
Comment on lines +58 to +60

@gemini-code-assist gemini-code-assist Bot Mar 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These commented-out lines and the preceding blank line should be removed. OTEL_SDK_DISABLED=true is already configured via ENV on line 52. Leaving commented-out code, especially for configuration, can cause confusion and clutters the Dockerfile. If these are for developer convenience, they should be documented elsewhere, like in a developer guide.

8 changes: 6 additions & 2 deletions kubernetes/kustomize/cornserve/overlays/dev/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,5 +29,9 @@ configMapGenerator:
- CORNSERVE_IMAGE_PREFIX=localhost:5000/cornserve
- CORNSERVE_IMAGE_PULL_POLICY=Always
- CORNSERVE_IMAGE_TAG=latest
# uncomment below to disable OTEL during benchmarking
# - OTEL_SDK_DISABLED=true
- OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=16384
- OTEL_SPAN_EVENT_COUNT_LIMIT=16384
- OTEL_SPAN_LINK_COUNT_LIMIT=16384
- OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT=16384
- OTEL_LINK_ATTRIBUTE_COUNT_LIMIT=16384
Comment on lines +32 to +36

@gemini-code-assist gemini-code-assist Bot Mar 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These OpenTelemetry settings are also duplicated in kubernetes/kustomize/cornserve/overlays/local/kustomization.yaml. To improve maintainability and avoid duplication, these common settings should be moved to the base kustomization (../../base). Both dev and local overlays include the base, so they would inherit these settings automatically. This will make future changes to these values easier to manage.


5 changes: 5 additions & 0 deletions kubernetes/kustomize/cornserve/overlays/local/kustomization.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,8 @@ configMapGenerator:
- CORNSERVE_IMAGE_PREFIX=docker.io/cornserve
- CORNSERVE_IMAGE_PULL_POLICY=Never
- CORNSERVE_IMAGE_TAG=latest
- OTEL_SPAN_ATTRIBUTE_COUNT_LIMIT=16384
- OTEL_SPAN_EVENT_COUNT_LIMIT=16384
- OTEL_SPAN_LINK_COUNT_LIMIT=16384
- OTEL_EVENT_ATTRIBUTE_COUNT_LIMIT=16384
- OTEL_LINK_ATTRIBUTE_COUNT_LIMIT=16384
Comment on lines +32 to +36

@gemini-code-assist gemini-code-assist Bot Mar 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

These OpenTelemetry settings are duplicated from the dev overlay. To adhere to the DRY (Don't Repeat Yourself) principle and improve maintainability, these common configurations should be defined once in the base kustomization (../../base) and inherited by all overlays.

2 changes: 1 addition & 1 deletion python/cornserve/task_executors/geri/executor/loader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ def get_registry_entry(model_id: str) -> tuple[RegistryEntry, PretrainedConfig |
class_name = model_index["_class_name"]
logger.info("Found pipeline class: %s", class_name)
except Exception:
logger.exception("Failed to load model_index.json from %s", model_id)
logger.warning("Failed to load model_index.json from %s", model_id)

@gemini-code-assist gemini-code-assist Bot Mar 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Changing from logger.exception to logger.warning is reasonable since failing to load model_index.json is a recoverable condition. However, this change loses the exception traceback, which can be crucial for debugging if the failure is due to an unexpected reason (e.g., file corruption, permission issues) rather than the file simply not existing. To retain this valuable debugging information while still logging at the WARNING level, you should add exc_info=True.

Suggested change
logger.warning("Failed to load model_index.json from %s", model_id)
logger.warning("Failed to load model_index.json from %s", model_id, exc_info=True)


# Then, if that didn't work, try to parse config.json for model_type instead
if class_name is None:
Expand Down
2 changes: 1 addition & 1 deletion third_party/vllm
Open in desktop
Submodule vllm updated 7 files
+1 −1 requirements/cuda.txt
+1 −1 vllm/entrypoints/openai/api_server.py
+34 −1 vllm/model_executor/models/qwen3_omni_moe_talker.py
+75 −28 vllm/v1/core/sched/scheduler.py
+30 −2 vllm/v1/engine/async_llm.py
+1 −1 vllm/v1/utils.py
+315 −99 vllm/v1/worker/gpu_model_runner.py