Openstreetmap search does not work on self-hosted but works on aws

[xhluca]

          @gasse there seems to be another openstreetmap issue. On AWS, the search works:

It appears that on self-hosted, it doesn't, as the /api endpoint does not seem to be set up properly.

Originally posted by @xhluca in #3 (comment)

It will affect certain tasks, such as webarena.257:

[xhluca]

Here's an example of what happens when the search is NOT working:

Here's the network traffic returning 404's:

XHRGET
https://my-cloudflare-webarena-hosted.com/way/474976514?xhr=1
[HTTP/2 404  87ms]

	
GET
	
scheme
	https
host
	my-cloudflare-webarena-hosted.com
filename
	/way/474976514
xhr
	1
Address
	172.64.80.1:443
Status
404
VersionHTTP/2
Transferred1.73 kB (241 B size)
Referrer Policystrict-origin-when-cross-origin
Request PriorityHighest
DNS ResolutionDNS over HTTPS

    	
    alt-svc
    	h3=":443"; ma=86400
    cache-control
    	no-cache
    cf-cache-status
    	DYNAMIC
    cf-ray
    	8f10aebec8b4a30a-YUL
    content-encoding
    	zstd
    content-language
    	en
    content-type
    	text/html; charset=utf-8
    date
    	Thu, 12 Dec 2024 21:07:11 GMT
    nel
    	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
    referrer-policy
    	strict-origin-when-cross-origin
    report-to
    	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=3wFnH2jywAk%2BpMNDb2OwkA%2FNWYpf7SGRjnMZm%2F2EiJg%2F%2BcXXVxIEQ%2BNotJKvddlgGHD2wHKzeczYJ5nyEH5zr%2BNIheaP0yBhnJ%2FGijPkN66n0nCVMXMy7aEIMZF9ojqZEcuwDEA%2BOTdjeZ%2Bj%2B%2Fge3EnyZZu5h6w%3D"}],"group":"cf-nel","max_age":604800}
    server
    	cloudflare
    server-timing
    	start_processing.action_controller;dur=0.17, cache_read.active_support;dur=0.05, sql.active_record;dur=2.15, render_partial.action_view;dur=0.23, render_template.action_view;dur=1.11, render_layout.action_view;dur=1.37, process_action.action_controller;dur=14.56, cache_write.active_support;dur=0.15
    server-timing
    	cfL4;desc="?proto=TCP&rtt=14714&min_rtt=3763&rtt_var=17346&sent=51&recv=78&lost=0&retrans=2&sent_bytes=11723&recv_bytes=8617&delivery_rate=1032112&cwnd=240&unsent_bytes=0&cid=204c455a595d2343&ts=20303&x=0"
    vary
    	Accept-Language
    x-content-type-options
    	nosniff
    x-download-options
    	noopen
    X-Firefox-Spdy
    	h2
    x-frame-options
    	sameorigin
    x-page-title
    	Not%20Found%20%7C%20OpenStreetMap
    x-permitted-cross-domain-policies
    	none
    x-request-id
    	d26506ec-a8b9-4474-8ee3-57f4a740993e
    x-runtime
    	0.049507
    x-xss-protection
    	1; mode=block
    	
    Accept
    	text/html, */*; q=0.01
    Accept-Encoding
    	gzip, deflate, br, zstd
    Accept-Language
    	en-CA,en-US;q=0.7,en;q=0.3
    Connection
    	keep-alive
    Cookie
    	_osm_session=b428b3b980dfabcb23bf1f5a3e0ea941; _osm_location=-79.90428|40.45578|19|M; _osm_directions_engine=fossgis_osrm_car
    DNT
    	1
    Host
    	my-cloudflare-webarena-hosted.com
    Priority
    	u=0
    Referer
    	https://my-cloudflare-webarena-hosted.com/way/474976514
    Sec-Fetch-Dest
    	empty
    Sec-Fetch-Mode
    	cors
    Sec-Fetch-Site
    	same-origin
    TE
    	trailers
    User-Agent
    	Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:133.0) Gecko/20100101 Firefox/133.0
    X-CSRF-Token
    	wLfQ9KGtWSim36quhJB2Zq3s73ssXECSEmw3lAY3pm5cu1uJDbny3DZH-yf8Rr9btB3-3jPumBjnEg-asOS9Gg
    X-Requested-With
    	XMLHttpRequest

[xhluca]

Here's the API endpoint that gets called:

https://my-cloudflare-webarena-hosted.com/api/0.6/way/474976514/full

Here's what it looks like:

On AWS instances, following the webarena official ami, e.g. on URLs like this:

http://ec2-random-numbers.compute-1.amazonaws.com:3000/way/474976514
http://ec2-random-numbers.compute-1.amazonaws.com:3000/api/0.6/way/474976514/full

It should appear like this:

[Tian99Yu]

I had the exact same issue. @xhluca have you managed to solve the problem?

[cherry979988]

I think the issue is that the map's postgres database was not populated automatically. I did the following and the issue seems to be solved.

• Install osmosis
  • sudo apt update
  • sudo apt install osmosis
• Download the osm dump
  • wget http://metis.lti.cs.cmu.edu/map_server_data/osm_dump.tar
  • tar xvf osm_dump.tar
• Add it to the postgres database

osmosis --read-pbf ~/osm_dump/us-northeast-latest.osm.pbf   --write-apidb host="localhost:54321" database="openstreetmap"   user="openstreetmap" password=" " validateSchemaVersion="no"

• While it runs, you can run the following to check if it’s really populating.
  • Go to ./openstreetmap-website
  • docker exec -it openstreetmap-website-db psql -U postgres openstreetmap -c "SELECT COUNT(*) FROM nodes;"
  • The number should be growing over time.
• The database population took 24 hours for me. You should get 175317503 nodes, 189052 relations and 19377165 ways, by running the SELECT COUNT command above.

cc: @xhluca @Tian99Yu

[Tian99Yu]

@cherry979988 Thank you for sharing!! This works for me

Just a side note, we better not reset / run experiment during the process of adding data to the database in openstreetmap-website-db container, ie: not run the experiments. I learnt it in the hard way and if you need happen to reset the container when psql is updating, the process would fail, and you need to

1. Remove the volumns linked to the containers, specificly: sudo docker volume rm openstreetmap-website_db-data , which is where the node data are stored
   To be safe, I also removed: sudo docker volume rm openstreetmap-website_web-storage, and sudo docker volume rm openstreetmap-website_web-tmp
2. remove the image to do a full reload: sudo docker image rm openstreetmap-website-db:latest and sudo docker image rm openstreetmap-website-web:latest
3. rebuild the image & container by running 01_docker_load_images.sh assuming the environment has been set up before. This will only rebuild the map related contianer, as others were left untouched.

After that, all will be reset.