Changing map routing server will affect the reward

[xhluca]

As discussed in #3, the tile server from CMU is down (it appears that the routing is still functioning, not sure about geocoding).

Apart from self-hosting (challenging and takes a lot of space), one approach is to use the official server. See this for more information on how to do that: web-arena-x/webarena#190

If you switch the routing URL to the official routing server yield different results. For example,

Official routing (reward=0)	Old routing (reward=1)

Note the difference: in the official routing (routing.openstreetmap.de), the route takes 10min, 5.2km and 10 steps. In the old routing (from cmu.edu's hosted routing server), it takes 9min, 5.1km, and 8 steps.

Here's the official answer:

  {
    "sites": [
      "map"
    ],
    "task_id": 155,
    "require_login": true,
    "storage_state": null,
    "start_url": "__MAP__",
    "geolocation": null,
    "intent_template": "What is the minimum travel time by car from {{location1}} to {{location2}}?",
    "instantiation_dict": {
      "location1": "Animal Rescue League of Pittsburgh",
      "location2": "Schenley park"
    },
    "intent": "What is the minimum travel time by car from Animal Rescue League of Pittsburgh to Schenley park?",
    "require_reset": false,
    "eval": {
      "eval_types": [
        "string_match"
      ],
      "reference_answers": {
        "fuzzy_match": [
          "9min"
        ]
      },
      "reference_url": "",
      "program_html": [],
      "string_note": "",
      "reference_answer_raw_annotation": "9min"
    },
    "intent_template_id": 36
  },

So it's something to consider before switching map server

[xhluca]

I would suggest changing the following:

webarena-setup/webarena/03_docker_create_containers.sh

Lines 16 to 51 in 55c9356

	# openstreetmap docker set up
	cd openstreetmap-website/
	# tile server URL (use default openstreetmap server)
	OSM_TILE_SERVER_URL="https://tile.openstreetmap.org/{z}/{x}/{y}.png"
	# geocoding server URL (use default openstreetmap server)
	OSM_GEOCODING_SERVER_URL="https://nominatim.openstreetmap.org/"
	# routing server URLs (use default openstreetmap server)
	OSM_ROUTING_SERVER_URL="https://routing.openstreetmap.de"
	OSM_CAR_SUFFIX="/routed-car"
	OSM_BIKE_SUFFIX="/routed-bike"
	OSM_FOOT_SUFFIX="/routed-foot"
	# original WebArena config (CMU server with different ports for each vehicule type)
	# OSM_ROUTING_SERVER_URL="http://metis.lti.cs.cmu.edu"
	# OSM_CAR_SUFFIX=":5000"
	# OSM_BIKE_SUFFIX=":5001"
	# OSM_FOOT_SUFFIX=":5002"
	# copy template files to be set up
	cp ../openstreetmap-templates/docker-compose.yml ./docker-compose.yml
	cp ../openstreetmap-templates/leaflet.osm.js ./vendor/assets/leaflet/leaflet.osm.js
	cp ../openstreetmap-templates/fossgis_osrm.js ./app/assets/javascripts/index/directions/fossgis_osrm.js
	# set up web server port
	sed -i "s|MAP_PORT|${MAP_PORT}|g" docker-compose.yml
	# set up tile server URL
	sed -i "s|url: 'https://tile.openstreetmap.org/{z}/{x}/{y}.png'|url: '${OSM_TILE_SERVER_URL}'|g" ./vendor/assets/leaflet/leaflet.osm.js
	# set up geocoding server URL
	sed -i "s|nominatim_url:.*|nominatim_url: \"$OSM_GEOCODING_SERVER_URL\"|g" ./config/settings.yml
	# set up routing server URLs
	sed -i "s|fossgis_osrm_url:.*|fossgis_osrm_url: \"$OSM_ROUTING_SERVER_URL\"|g" ./config/settings.yml
	sed -i "s|__OSMCarSuffix__|${OSM_CAR_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	sed -i "s|__OSMBikeSuffix__|${OSM_BIKE_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	sed -i "s|__OSMFootSuffix__|${OSM_FOOT_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
	docker compose create

Into this:

# MAP CONTAINER SETUP STARTS HERE

cd openstreetmap-website/
cp ../openstreetmap_docker-compose.template.yml docker-compose.yml

# replace the ports in the docker-compose file using sed
sed -i "s|MAP_PORT|${MAP_PORT}|g" docker-compose.yml
sed -i "s|MAP_DB_PORT|${MAP_DB_PORT}|g" docker-compose.yml
# replace the name "openstreetmap_web" with the container name in $OPENSTREETMAP_WEB_CONTAINER_NAME
sed -i "s|openstreetmap_web|${OPENSTREETMAP_WEB_CONTAINER_NAME}|g" docker-compose.yml
# replace the name "openstreetmap_db" with the container name in $OPENSTREETMAP_DB_CONTAINER_NAME
sed -i "s|openstreetmap_db|${OPENSTREETMAP_DB_CONTAINER_NAME}|g" docker-compose.yml

# The following are the default URLs
OSM_TILE_SERVER_URL="https://tile.openstreetmap.org/{z}/{x}/{y}.png"
OSM_ROUTING_SERVER_URL="https://routing.openstreetmap.de/"  # trailing / allows for routed-<name> to be appended
OSM_GEOCODING_SERVER_URL="https://nominatim.openstreetmap.org/"

# The following are the original URLs released by CMU.
WA_TILE_SERVER_URL="http://ogma.lti.cs.cmu.edu:8080/tile/{z}/{x}/{y}.png"
WA_ROUTING_SERVER_URL="http://metis.lti.cs.cmu.edu:"  # trailing : allows for port to be appended
WA_GEOCODING_SERVER_URL="http://metis.lti.cs.cmu.edu:8085/"

# We use the OSM tile server since the webarena server is not available. However, we use the
# routing and geocoding servers from webarena, since the OSM version was updated since webarena's release
# which means it will have different paths and time estimate, thus skewing the results.
# See: https://github.com/gasse/webarena-setup/issues/4
TILE_SERVER_URL=$OSM_TILE_SERVER_URL
ROUTING_SERVER_URL=$WA_ROUTING_SERVER_URL
GEOCODING_SERVER_URL=$WA_GEOCODING_SERVER_URL

cp ../leaflet.osm.template.js ./vendor/assets/leaflet/leaflet.osm.js
sed -i "s|url: 'https://tile.openstreetmap.org/{z}/{x}/{y}.png'|url: '${TILE_SERVER_URL}'|g" ./vendor/assets/leaflet/leaflet.osm.js
sed -i "s|.*fossgis_osrm_url:.*|fossgis_osrm_url: \"$ROUTING_SERVER_URL\"|g" ./config/settings.yml
sed -i "s|.*nominatim_url:.*|nominatim_url: \"$GEOCODING_SERVER_URL\"|g" ./config/settings.yml

# use the following text for the routing servers, e.g. cloudflare or official
OSM_CAR_SUFFIX="routed-car"
OSM_BIKE_SUFFIX="routed-bike"
OSM_FOOT_SUFFIX="routed-foot"

# alternatively, use ports, e.g. for webarena's setup
WA_CAR_SUFFIX=5000
WA_BIKE_SUFFIX=5001
WA_FOOT_SUFFIX=5002

# We use the webarena suffixes since we are using the webarena routing server
MAP_CAR_SUFFIX=$WA_CAR_SUFFIX
MAP_BIKE_SUFFIX=$WA_BIKE_SUFFIX
MAP_FOOT_SUFFIX=$WA_FOOT_SUFFIX

# now copy the fossgis_osrm.template.js to fossgis_osrm.js and replace __OSMCarSuffix__ with $OSM_CAR_SUFFIX
cp ../fossgis_osrm.template.js ./app/assets/javascripts/index/directions/fossgis_osrm.js

# replace the suffixes in the fossgis_osrm.js file
sed -i "s|__OSMCarSuffix__|${MAP_CAR_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
sed -i "s|__OSMBikeSuffix__|${MAP_BIKE_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js
sed -i "s|__OSMFootSuffix__|${MAP_FOOT_SUFFIX}|g" ./app/assets/javascripts/index/directions/fossgis_osrm.js

sudo docker compose create

[xhluca]

For cloudflare, one needs to do a redirect. here's some instructions:

source 00_vars.sh

WA_ROUTING_CF_NAME_CAR="cmu-redirect-routing-car"
WA_ROUTING_CF_NAME_BIKE="cmu-redirect-routing-bike"
WA_ROUTING_CF_NAME_FOOT="cmu-redirect-routing-foot"

WA_ROUTING_URL_CAR="http://metis.lti.cs.cmu.edu:5000"
WA_ROUTING_URL_BIKE="http://metis.lti.cs.cmu.edu:5001"
WA_ROUTING_URL_FOOT="http://metis.lti.cs.cmu.edu:5002"

screen -S $WA_ROUTING_CF_NAME_CAR -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_CAR $WA_ROUTING_URL_CAR
screen -S $WA_ROUTING_CF_NAME_BIKE -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_BIKE $WA_ROUTING_URL_BIKE
screen -S $WA_ROUTING_CF_NAME_FOOT -dm bash create_and_run_redirect.sh $WA_ROUTING_CF_NAME_FOOT $WA_ROUTING_URL_FOOT

Content of create_and_run_redirect.sh:

CF_DOMAIN="your-cloudflare-domain.com"

create_route_and_run_redirect() {
    CF_TUNNEL_NAME=$1
    CF_TUNNEL_TARGET_URL=$2

    # delete the tunnel if it exists, and recreate it
    echo "Cleaning up tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel cleanup $CF_TUNNEL_NAME
    echo "Deleting tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel delete $CF_TUNNEL_NAME
    echo "Creating tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel create $CF_TUNNEL_NAME

    # route and start the tunnel
    echo "Routing tunnel: $CF_TUNNEL_NAME"
    cloudflared tunnel route dns --overwrite-dns $CF_TUNNEL_NAME "${CF_TUNNEL_NAME}.${CF_DOMAIN}"
    echo "Running tunnel ${CF_TUNNEL_NAME} at URL: https://${CF_TUNNEL_NAME}.${CF_DOMAIN}"
    echo "Target URL: $CF_TUNNEL_TARGET_URL"
    cloudflared tunnel run --url $CF_TUNNEL_TARGET_URL $CF_TUNNEL_NAME
}

create_route_and_run_redirect $1 $2

[jayl940712]

Hi all, I know this issue have been quite stale. I also encountered the exact problem and it turns out the fix is to self host geocoding and routing. It is still best to use openstreetmap for tile service because self hosting is very slow for it.

Webarena updated a yaml for self hosting map services in aws, which can be easily converted to a bash script. bash script.

Below is the final config I used for 03 script:

OSM_TILE_SERVER_URL="https://tile.openstreetmap.org/{z}/{x}/{y}.png"

MAP_HOSTNAME="YOUR_HOSTED_MAP_SERVICE_IP"
OSM_GEOCODING_SERVER_URL="http://${MAP_HOSTNAME}:8085/"
OSM_ROUTING_SERVER_URL="http://${MAP_HOSTNAME}"
OSM_CAR_SUFFIX=":5000"
OSM_BIKE_SUFFIX=":5001"
OSM_FOOT_SUFFIX=":5002"

I think we can close this issue @xhluca @gasse