python-runtime.md

Run inference with the Python runtime

Use the Edge-LLM C++ runtime through Python to generate text from local model files. This example uses the wheel without optional dependencies. For chat or HTTP serving, follow the trtedge CLI guide.

Prepare your environment and model

Complete Install without optional dependencies and keep that environment activated for the commands below.

Provide a complete local Qwen3.5-0.8B Safetensors checkpoint, including tokenizer.json. The example does not download model files. Replace /path/to/Qwen3.5-0.8B with your checkpoint directory.

Build an engine and generate text

1. {download}Download the example script <../../../../examples/python/installed_wheel_build_and_infer.py> and save it outside a source checkout.
2. Choose an output directory for the generated engine. The script deletes and replaces this directory, so use a disposable directory.
3. Run the script, replacing its path and the checkpoint path below:

python -I /path/to/installed_wheel_build_and_infer.py \
  /path/to/Qwen3.5-0.8B /tmp/edgellm-base-engines \
  --workflow base

The script builds a text engine, runs a prompt through tensorrt_edgellm.runtime.LLMRuntime, and checks for generated text and tokens. On success, it prints the result and INSTALLED_WHEEL_BUILD_AND_INFERENCE_PASSED.

For more engine-build options, see the direct builder guide.

Use an existing engine

If you already have a compatible text engine and its local checkpoint, save the following code as run_engine.py. Replace the checkpoint and engine paths in the final line. This example runs inference without rebuilding the engine.

---
language: python
pyobject: _infer_base
prepend: |
  from pathlib import Path
  from typing import Tuple
append: |
  print(_infer_base(Path("/path/to/checkpoint"), Path("/path/to/engines"), "Hello", 32))
---

Run it in the same environment:

python /path/to/run_engine.py

Successful execution prints the generated text and token count.

C++ example executables require a source installation.