Skip to content
BranchesTags
 
 

Repository files navigation

Jacob's Ladder

Welcome, it's recommended that you start HERE, as this main README will allow you to navigate this large and complex project. Each of the subsections have their own respective READMEs and docs for full explanations.

Jacob's Ladder is a modular, system-agnostic UAV command-and-control (C2) framework built on ROS 2 Humble and PX4. It lets you write autonomous drone missions entirely in ROS 2 — the same code runs in Gazebo simulation on your laptop and on real hardware in the field, with no rewrites needed.

The framework uses PX4 external modes instead of the traditional offboard API. External modes register directly with PX4 through the companion computer, appearing as selectable flight modes in QGroundControl alongside the built-in ones (Stabilized, Position, Mission, etc.). If the companion computer ever stops communicating, PX4 automatically failsafes — so the system is safe by design.

Jacob's Ladder Structure

Within the project you'll find several pacakges, this README serves as a higher level overview of all the components involved in the project. For a deeper view, there are embedded markdown files which break down specific rationale, processes, and objectives.

Python Environment

Use the repo uv environment for Python dependencies. It is created with system site packages enabled so ROS 2, cv_bridge, JetPack OpenCV, and other system Python modules remain visible inside the venv.

uv venv --python 3.10 --system-site-packages .venv
uv sync
source .venv/bin/activate
source /opt/ros/humble/setup.bash
source install/setup.bash  # after colcon build

That's it — uv sync installs everything, including the YOLO/drogue image-processing stack (torch, torchvision, ultralytics). There is no extra to enable and no bootstrap script to run.

opencv-python is deliberately not installed — see OpenCV on the Jetson below.

On JetPack 6.2 / CUDA 12.6 (aarch64), uv sync pulls the pinned CUDA-enabled torch and torchvision wheels from pypi.jetson-ai-lab.io/jp6/cu126 plus the jetson-cudss-bootstrap workspace package (see packages/jetson_cudss_bootstrap), which preloads the cuDSS runtime shared libraries torch needs at import time. No LD_LIBRARY_PATH exports or extra install steps are required — uv sync (or uv run ...) is enough.

The only reason to pass --system-site-packages when creating the venv is so ROS 2, cv_bridge, and JetPack OpenCV stay visible inside it. If you re-create the venv, keep that flag.

OpenCV on the Jetson

The OpenCV that ships with the ARK Electronics / JetPack image cannot build or run this workspace. You must build the CUDA-enabled one with installation_scripts/install_opencv.sh.

A stock ARK JetPack 6.2 image has three OpenCVs on it, and none of them work:

Install Version Location CUDA contrib aruco
Ubuntu debs (ROS Humble links these) 4.5.4d /usr/lib/aarch64-linux-gnu no no
JetPack/ARK nvidia-opencv 4.8.0 /usr/lib, headers in /usr/include/opencv4 no no
opencv-python pip wheel 4.9.0.80 venv site-packages no n/a

Three concrete problems:

  1. No CUDA. /usr/include/opencv4/opencv2/cvconfig.h contains /* #undef HAVE_CUDA */, and there is no libopencv_cuda*.so anywhere on the image. cv2.cuda.getCudaEnabledDeviceCount() returns 0.
  2. No contrib modules, so aruco_tracker cannot compile. It does find_package(OpenCV 4 REQUIRED COMPONENTS core imgproc calib3d aruco) and includes <opencv2/aruco.hpp>. The JetPack build only has the core-repo subset (opencv2/objdetect/aruco_detector.hpp) — no aruco.hpp, no libopencv_aruco.
  3. A broken pkg-config file. /usr/lib/pkgconfig/opencv4.pc from nvidia-opencv-dev declares prefix=/usr/local, but nothing OpenCV is installed there. Any pkg-config-driven build gets -L/usr/local/lib -lopencv_core and fails to link. Installing to /usr/local (the default below) incidentally makes that file correct again.

Building it

REMOVE_DEFAULT_OPENCV=no OPENCV_BUILD_JOBS=6 ./installation_scripts/install_opencv.sh

Takes roughly 60–120 minutes on an 8-core Orin NX and needs ~10 GB of disk. It needs sudo for the apt step and make install, so run it interactively.

Answer no to the "remove the default OpenCV" prompt (or pass REMOVE_DEFAULT_OPENCV=no as above). apt purge *libopencv* would take ros-humble-cv-bridge, ros-humble-image-geometry and the rest of image_pipeline with it, since they link libopencv_*.so.4.5d.

Useful environment variables:

Variable Default Notes
OPENCV_VERSION 4.10.0
OPENCV_CUDA_ARCH 8.7 Orin NX / Orin Nano / AGX Orin are all sm_87. Use 7.2 for Xavier. Listing only your arch roughly halves build time.
OPENCV_WITH_CUDNN ON Set OFF if cmake fails to detect cuDNN. JetPack 6.2 ships cuDNN 9, which older OpenCV releases mis-detect. Nothing here needs it — YOLO runs inference through torch, not cv2.dnn.
OPENCV_BUILD_JOBS nproc - 1 Lower it if the build gets OOM-killed.
OPENCV_PYTHON_VENV repo .venv Where the cv2 bindings are installed.
OPENCV_INSTALL_PREFIX /usr/local

The script verifies itself at the end: it checks that the cv2 the venv actually imports is the version just built and that cv2.cuda.getCudaEnabledDeviceCount() >= 1, and exits non-zero otherwise. If it reports a different version than it just built, something on sys.path is shadowing the build — almost always the opencv-python pip wheel (uv pip uninstall opencv-python) or the system python3-opencv deb.

Re-running the script is incremental: it keeps ~/opencv_src and its release/ build directory, so a second run only compiles what changed. Set OPENCV_CLEAN_BUILD=1 to force a full rebuild.

The script installs the cv2 Python bindings directly into the repo venv's site-packages, because OpenCV's default ${prefix}/lib/python3.10/site-packages is invisible to a venv. Venv site-packages precedes /usr/lib/python3/dist-packages on sys.path, so this build wins import cv2 over the system 4.5.4 deb.

Why opencv-python is not a dependency

The pip wheel is CPU-only and lives in the same site-packages the script writes cv2 into, so it shadows the CUDA build and silently disables GPU acceleration for every cv2 call. It is therefore excluded from pyproject.toml twice over: it is not a direct dependency, and because ultralytics requires it unconditionally, [tool.uv] override-dependencies drops the transitive requirement with a never-true marker.

If you are working on a machine without the CUDA build, install it by hand:

uv pip install opencv-python

Building the workspace against it

Tell colcon which OpenCV to use, so you don't get the JetPack 4.8.0 by accident:

colcon build --cmake-args -DOpenCV_DIR=/usr/local/lib/cmake/opencv4

The workspace vendors src/vision_opencv, so cv_bridge and image_geometry are rebuilt against the same OpenCV as everything else. Make sure install/setup.bash is sourced after /opt/ros/humble/setup.bash so the overlay shadows the 4.5-linked debs — loading two OpenCV ABIs into one process causes crashes that look random.

START HERE:

Package Objective
Micro-XRCE-DDS-Agent Should NOT be Altered : Use given branch -> Communication Agent from Drone to Host Computer
aruco_tracker ROS 2 Wrapper for OpenCV detection of an ArUco Marker
drogue_flight TODO: In-Progress Porting for flying to a detected KC-130 drogue
example_autonomous_mode Start here to write a new mode -> A minimal, working External Mode (take off, hold, land) documented line by line and meant to be copied
jacob_manual ROS 2 External Modes that Require Manual Control to get in the air, but fly autonomous missions after
precision_land ROS 2 External Modes that fly autonomous missions for object detection, trajectory planning, and landing
oak_d_visual_odometry ROS 2 nodes for OAK-D visual odometry using NVIDIA cuVSLAM, with optional PX4 VehicleOdometry output for flight.
px4-ros2-interface-lib Should NOT be Altered : Use Given Branch -> ROS 2 <-> PX4 Bridge, allows us to create External Modes
px4_msgs Should NOT be Altered : Use Given Branch -> PX4 Messages for ROS 2 Communication
px4_msgs_old Should NOT be Altered : Use Given Branch -> More PX4 Messages for ROS 2 Communication
translation_node Should NOT be Altered : Use Given Branch -> Introduced in PX4 v1.15.0, translates old px4 messages necessary for past versions to new px4 messages
vision_opencv Should NOT be Altered : Use Given Branch -> OpenCV STD library

Not a ROS 2 package, but it runs on the vehicle:

Component Objective
battery_monitor Watches pack voltage on the INA238 and alerts at configurable land / min thresholds. Runs as a systemd service, independent of ROS 2.

What Can It Do?

Capability Description
Precision Landing Detect an ArUco marker with a downward camera, approach it, and autonomously land on it — even on a moving platform
Front-Camera Approach Fly toward a target detected by a forward-facing camera using PID control
Combined Approach + Land Use a front camera to approach, then seamlessly hand off to a downward camera for precision landing
Drogue Collection Detect a drogue target via YOLO, plan a smooth S-curve trajectory, and fly to it
GPS-Denied Flight (VIO) Estimate position indoors with an OAK-D stereo camera and NVIDIA cuVSLAM, fed to PX4's EKF2 as external vision — see oak_d_visual_odometry
Takeoff & Hold Simple building-block modes for taking off, holding altitude, and landing
Battery Voltage Alerting Watch the pack on the INA238 and warn — desktop popup, terminal broadcast — at your configured land and minimum voltages, before a pack gets over-drained. See battery_monitor

If You're New to ROS 2

A few concepts that will help you navigate this project:

  • Node — A single process that does one job (e.g. track ArUco markers, send flight commands). Nodes communicate by publishing and subscribing to topics.
  • Topic — A named data channel. One node publishes messages to a topic; other nodes subscribe to receive them. For example, the ArUco tracker publishes marker positions to a topic that the precision landing mode subscribes to.
  • Package — A folder containing related code, a CMakeLists.txt (or setup.py for Python), and a package.xml manifest. Each folder under src/ is a package.
  • Launch file — A Python script that starts multiple nodes at once with the right parameters. You'll find these in each package's launch/ folder.
  • Workspace — The overall project directory. You build everything with colcon build and source install/setup.bash to make your nodes available.

How the Pieces Fit Together

┌───────────────────────────────────────────────────────────┐
│                     QGroundControl                        │
│              (select External Mode here)                  │
└───────────────────┬───────────────────────────────────────┘
                    │ MAVLink
┌───────────────────▼───────────────────────────────────────┐
│                   PX4 Autopilot                           │
│            (flight controller firmware)                   │
└───────────────────┬───────────────────────────────────────┘
                    │ uXRCE-DDS (serial or UDP)
┌───────────────────▼───────────────────────────────────────┐
│              Micro XRCE-DDS Agent                         │
│         (bridges PX4 topics ↔ ROS 2 topics)               │
└───────────────────┬───────────────────────────────────────┘
                    │ ROS 2 topics
       ┌────────────┼────────────────────┐
       │            │                    │
       ▼            ▼                    ▼
┌─────────────┐ ┌──────────┐ ┌────────────────────┐
│ Translation │ │  ArUco   │ │   External Mode    │
│    Node     │ │ Tracker  │ │ (e.g. PrecisionLand│
│             │ │          │ │  TakeoffHold, etc.) │
│ Converts    │ │ Detects  │ │                    │
│ PX4 msg     │ │ markers  │ │ Reads sensor data, │
│ versions    │ │ from     │ │ runs state machine,│
│             │ │ camera   │ │ sends setpoints    │
└─────────────┘ └──────────┘ └────────────────────┘

Prerequisites

  • Ubuntu Linux (22.04 recommended)
  • Docker Engine
  • PX4 Autopilot (v1.16.0 recommended)
  • QGroundControl Daily Build

Quick Start

1. Clone PX4

Create your workspace on your system and add in PX4-Autopilot

mkdir -p ~/jacob_ladder_ws/src && cd ~/jacob_ladder_ws/src
git clone https://github.com/PX4/PX4-Autopilot.git --recursive

The v1.16.0 tag is highly recommended. The included translation node provides compatibility with most PX4 versions from 1.16 onward, but some versions may lack the Gazebo worlds and models needed for simulation — refer to PX4's official documentation if that's the case.

cd PX4-Autopilot 
git status
git checkout v1.16.0

2. Install QGroundControl

Download the daily build: https://docs.qgroundcontrol.com/master/en/qgc-user-guide/releases/daily_builds.html

3. Clone Jacob's Ladder

git clone https://github.com/CursedRock17/Jacob_Ladder.git
cd Jacob_Ladder
git checkout drogue_collector
git submodule update --init --recursive

That is the whole clone procedure — there is nothing to patch by hand.

About the px4-ros2-interface-lib submodule

It points at cdenihan/px4-ros2-interface-lib, a fork of Auterion/px4-ros2-interface-lib, on branch jacob-ladder/executor-skip-msg-check. The fork is the upstream 1.6.0 tag plus a single commit adding setSkipMessageCompatibilityCheck() and its backing flag to ModeExecutorBase.

Upstream only provides that method on ModeBase, where it is protected — and ModeExecutorBase's friend relationship with ModeBase is not inherited by derived classes, so no executor subclass can reach it. Several of our mode executors call it, so without the fork precision_land, drogue_flight and jacob_manual fail to compile with:

error: 'setSkipMessageCompatibilityCheck' was not declared in this scope

This used to be distributed as a patch.diff in the repo root that you applied by hand. That is gone: it was undocumented, any git submodule update silently reverted it, and it left the submodule permanently dirty. The change now lives in a real commit, so a plain git submodule update --init --recursive gives you a tree that builds.

To rebase onto a newer upstream release:

cd src/px4-ros2-interface-lib
git remote add upstream https://github.com/Auterion/px4-ros2-interface-lib.git
git fetch upstream --tags
git rebase <new-tag>            # one commit to replay
git push fork HEAD --force-with-lease
cd ../.. && git add src/px4-ros2-interface-lib && git commit

Note the guard short-circuits waitForFMU() as well as messageCompatibilityCheck(), so an executor that skips the check also does not wait for the FMU before registering.

4. Set Up Docker

The docker/ directory contains a layered Dockerfile stack built on px4io/px4-dev-base-jammy. Pre-built images are available on Docker Hub, so you can skip manual dependency installation entirely.

Install Docker Engine: https://docs.docker.com/engine/install/ubuntu/

Start the container:

If on Linux: First make sure your user has docker permissions, a quick check:

sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
# Make sure to refresh computer for groups to kick into effect
docker run hello-world

Then open up the container

xhost +
export CONTAINER_NAME=your_container_name

docker run -it --privileged \
  -u $(id -u):$(id -g) \ 
  -v ~/src/PX4-Autopilot:/src/PX4-Autopilot/:rw \
  -v ~/path/to/Jacob_Ladder:/path/to/Jacob_Ladder/:rw \
  -v /tmp/.X11-unix:/tmp/.X11-unix:ro \
  -e DISPLAY=:0 \
  --network host \
  --name=$CONTAINER_NAME \
  jacobsafeer/px4-dev-harmonic-jammy-humble-opencv-rqt:latest bash

If on Windows:

export CONTAINER_NAME=your_container_name
docker run -it --privileged \
    --user root \
    -v "C:\path\PX4":/src/PX4/:rw \
    -v "C:\path\Jacob_Ladder":/src/Jacob_Ladder/:rw \
    -e DISPLAY=:0 \
    --network host \
    --name $CONTAINER_NAME jacobsafeer/px4-dev-harmonic-jammy-humble-opencv-rqt:latest bash

Not tested on MacOS Maybe If you running a windowing system other than X11 or notably running simulation with a GPU you may have to make -e and -v alterations to the the windowing and display sections to allow it run.

Replace /path/to with your actual path to Jacob_Ladder, and $CONTAINER_NAME with whatever you'd like.

If you're not already in the container, start and enter

docker start $CONTAINER_NAME
docker exec -it $CONTAINER_NAME bash

When inside the container, build the workspace:

cd /path/to/Jacob_Ladder
source /opt/ros/humble/setup.bash
colcon build

5. Run the Example Simulation

Customize a launch script using the template in the existing launch_scripts/ directory. Each script opens multiple terminal tabs — one for PX4 SITL, one for the DDS agent, one for each ROS node — so everything starts together.

# On the host machine
./QGroundControl.AppImage           # Start QGC
docker start $CONTAINER_NAME        # Start the container
./launch_scripts/precision_land.sh  # Run the launch script

Select the precision landing mode from the QGroundControl dropdown:

Other Flight Mode Scripts

precision_land.sh is just one of several ready-to-run scripts in launch_scripts/. Each one starts PX4 SITL in a specific Gazebo world and launches the ROS nodes needed for that flight mode. Pick whichever matches what you want to test.

Script Flight Mode Gazebo World Purpose
precision_land.sh precision_land/front_to_precision_land.launch.py gz_x500_dual_cam_aruco_dual_ids Front-camera approach, then downward precision landing on an ArUco tag
approach_aruco.sh precision_land/front_approach.launch.py gz_x500_dual_cam_aruco_dual_ids Front-camera ArUco tag approach only (no landing)
jacob_manual.sh jacob_manual/front_approach.launch.py gz_x500_dual_cam_aruco_dual_ids Same as approach_aruco.sh, but runs the newer jacob_manual package version
takeoff_hover.sh precision_land/takeoff_hold.launch.py gz_x500_dual_cam Takeoff and hold indefinitely — good for hover tuning
takeoff_hover_land.sh precision_land/takeoff_land.launch.py gz_x500_dual_cam Takeoff, hold briefly, then land — simplest end-to-end mode
offboard_blank.sh precision_land/blank_mode.launch.py gz_x500_dual_cam Empty external-mode template — registers with PX4 and does nothing, for when you already know the framework
moving_launch.sh precision_land/track_follow.launch.py Custom moving platform world Track and land on a moving ArUco platform (uses the v1_16_tracker aruco launch)
real_launch.sh (edit before use) Real hardware DDS agent + RViz shell for Jetson/Pixhawk flight testing — all mode commands are commented out so you can uncomment the one you want
fake_drogue.sh (no flight mode) Real hardware Drogue camera feed sanity check — runs the DDS agent and ros2 topic hz on the camera topic

Writing a new mode rather than running an existing one? Start from example_autonomous_mode — a minimal mode that actually flies (take off, hold, land), documented line by line, with its own launch file:

ros2 launch example_autonomous_mode example_autonomous_mode.launch.py

Every script follows the same shape: the first tab is PX4 SITL (or omitted for real hardware), then the DDS agent, the translation node, any trackers, and the mode itself. If a script doesn't match your container name or PX4 version, open it and edit the container_name variable at the top — it's the same in every script.

Paths: no script hardcodes a home directory. Each one sources jl_env.sh from the repo root, which derives the workspace root from its own location, so a clone works under any username at any path. Override any of these in the environment when the defaults don't fit:

Variable Default
JL_WS_ROOT directory containing jl_env.sh (this repo)
JL_PX4_DIR PX4-Autopilot beside or inside the workspace, else ~/PX4-Autopilot
JL_ROS_DISTRO $ROS_DISTRO, else humble
JL_VENV $JL_WS_ROOT/.venv
JL_DOCKER_WS_DIR / JL_DOCKER_PX4_DIR the host paths above — set these if the container mounts them elsewhere
JL_TRACKTOR_BEAM_DIR tracktor-beam beside the workspace, else under $HOME
JL_DROGUE_MODEL YOLO weights from the installed drogue_flight share dir, else the source tree

The systemd units are templates (services/*.service.in) rendered by ./services/install_services.sh, which fills in the real workspace path and owning user for the machine it runs on.

Launch Script Options

Tab Options
precision_land precision_land.launch.py (default ArUco world) or track_follow.launch.py (moving platform world)
aruco_tracker aruco_tracker.launch.py (PX4 1.15.x), v1_16_tracker.launch.py (PX4 1.16.x default), or moving_aruco.launch.py (PX4 1.16.x moving platform)

Features in the repository

For more detailed tutorials checkout the general_docs section of the project which contains information on how to:

That way you have access to more in-depth tutorials.

Also, all of the code in the repository can be totalled expanded.

Flight Parameters

Different configurations of physical drones may require different PX4 Parameter configurations any and all tested forms of parameters are labelled in the config/params directory.

Before You Fly

  1. Manual flight first — Get the drone flying reliably in Stabilized and Position modes before running any autonomy
  2. Networked sim test — Run PX4 SITL on your PC and the ROS nodes on the Jetson over the same network to verify communication
  3. Props-off test — Run your flight mode with propellers removed and verify PX4 receives setpoints via QGroundControl
  4. Verify topics — Confirm PX4 topics are visible: ros2 topic list should show /fmu/out/vehicle_odometry, /fmu/out/vehicle_status, etc.

Additional Resources

Questions

Email: lwendlan@umd.edu

License

See LICENSE for details.

About

No description, website, or topics provided.

Resources

Readme
License
Activity

Stars

3 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages