Backend instability leading to 500 errors

[geekdave]

Summary

The backend becomes responsive, causing requests to the /word endpoint to fail with a 500 error, leading to the UI showing ... instead of words.

Steps to reproduce:

1. Deploy stack using setup instructions: https://github.com/WordCannon/deploy/blob/master/README.md
2. Tail service log file using docker service logs --follow wordcannon_app
3. Open UI at http://localhost and start watching words go by
4. View logs:
   At first everything is going well:

   wordcannon_app    | Firing cannon on http://0.0.0.0:8080 (Vowel trigger count 4)
   wordcannon_app    | /word 200 [248 ms] differential (5 vowels)
   wordcannon_app    | /word 200 [286 ms] designation (5 vowels)
   wordcannon_app    | /word 200 [344 ms] fun (1 vowels)
   wordcannon_app    | /word 200 [358 ms] returns (2 vowels)
   wordcannon_app    | /word 200 [160 ms] contamination (6 vowels)
   

5. Then suddenly at some point the 200s change to 500s and words stop being retrieved:

   wordcannon_app    | /word 200 [371 ms] conventions (4 vowels)
   wordcannon_app    | /word 500 [160 ms] ??? (0 vowels)
   wordcannon_app    | /word 500 [129 ms] ??? (0 vowels)
   wordcannon_app    | /word 500 [277 ms] ??? (0 vowels)
   wordcannon_app    | /word 500 [368 ms] ??? (0 vowels)
   wordcannon_app    | /word 500 [362 ms] ??? (0 vowels)    
   

6. Forcing a restart of the service fixes it temporarily: docker service update --force wordcannon_app

Next Steps

Backend team is promising a fix soon. While they are on the case, let's not step on their toes and try to fix the actual bug in the NodeJS code.

Instead, we need the backend service to be as resilient as possible even with the bug. The ops team is getting tired of logging into the server and manually restarting the service all the time.

Until this bug is fixed, we should investigate using a solution to automatically restart the service if the wordcannon service on port 8080 starts serving 500 errors.