README.md

cua-computer is a Computer-Use Interface (CUI) framework powering Cua for interacting with local macOS and Linux sandboxes, PyAutoGUI-compatible, and pluggable with any AI agent systems (Cua, Langchain, CrewAI, AutoGen). Computer relies on Lume for creating and managing sandbox environments.

Get started with Computer

from computer import Computer

computer = Computer(os_type="macos", display="1024x768", memory="8GB", cpu="4")
try:
    await computer.run()
    
    screenshot = await computer.interface.screenshot()
    with open("screenshot.png", "wb") as f:
        f.write(screenshot)
    
    await computer.interface.move_cursor(100, 100)
    await computer.interface.left_click()
    await computer.interface.right_click(300, 300)
    await computer.interface.double_click(400, 400)

    await computer.interface.type("Hello, World!")
    await computer.interface.press_key("enter")

    await computer.interface.set_clipboard("Test clipboard")
    content = await computer.interface.copy_to_clipboard()
    print(f"Clipboard content: {content}")
finally:
    await computer.stop()

Install

To install the Computer-Use Interface (CUI):

pip install "cua-computer[all]"

The cua-computer PyPi package pulls automatically the latest executable version of Lume through pylume.

Run

Refer to this notebook for a step-by-step guide on how to use the Computer-Use Interface (CUI):

• Computer-Use Interface (CUI)

Using the Gradio Computer UI

The computer module includes a Gradio UI for creating and sharing demonstration data. We make it easy for people to build community datasets for better computer use models with an upload to Huggingface feature.

# Install with UI support
pip install "cua-computer[ui]"

Note: For precise control of the computer, we recommend using VNC or Screen Sharing instead of the Computer Gradio UI.

Building and Sharing Demonstrations with Huggingface

Follow these steps to contribute your own demonstrations:

1. Set up Huggingface Access

Set your HF_TOKEN in a .env file or in your environment variables:

# In .env file
HF_TOKEN=your_huggingface_token

2. Launch the Computer UI

# launch_ui.py
from computer.ui.gradio.app import create_gradio_ui
from dotenv import load_dotenv
load_dotenv('.env')

app = create_gradio_ui()
app.launch(share=False)

For examples, see Computer UI Examples

3. Record Your Tasks

View demonstration video

RecordTrajectory.mp4

Record yourself performing various computer tasks using the UI.

4. Save Your Demonstrations

View demonstration video

SaveDemonstration.mp4

Save each task by picking a descriptive name and adding relevant tags (e.g., "office", "web-browsing", "coding").

5. Record Additional Demonstrations

Repeat steps 3 and 4 until you have a good amount of demonstrations covering different tasks and scenarios.

6. Upload to Huggingface

View demonstration video

UploadHuggingface.mp4

Upload your dataset to Huggingface by:

• Naming it as {your_username}/{dataset_name}
• Choosing public or private visibility
• Optionally selecting specific tags to upload only tasks with certain tags

Examples and Resources

• Example Dataset: ddupont/test-dataset
• Find Community Datasets: 🔍 Browse CUA Datasets on Huggingface