PR bot: scrape, summarize build logs

[justinmk]

PR bot: goals

1. Get the build status of the last 20(?) recently-active PRs.
2. Gather/parse/summarize the build logs.
3. Leave a comment on the PR or update existing comment.
   • Example
   • Summarize the current build results.
     • Provide direct links to the "raw" logs for appveyor/travis/etc
   • Never adds more than 1 comment on a PR (update existing comment with new info).

Other ideas

• Generate a page showing the number of tests executed on the most recent build of neovim/neovim master. bot-ci could use GitHub/Travis APIs to pull the build logs of the latest build of neovim master, then update a simple txt or html file in the docs repo which shows the total number of unit tests and functional tests.
  • Example:

  unit tests: 200 successes / 0 failures / 0 errors / 0 pending : 0.0 seconds
  functional tests: 233 successes / 0 failures / 0 errors / 2 pending : 0.0 seconds
  

[fwalch]

We could also create a graph that shows the respective number of successful/failed/.. tests over time, so we can immediately see if the numbers go down.

[elmart]

All tests should pass on every build, so... numbers can't go down...

[justinmk]

@elmart But if tests are erroneously marked as pending() because of possibly faulty logic, or if a describe() or it() block is accidentally out of place, it could cause a test to fall off the radar and we might not notice for a long time (coverity might show a .1% decrease, which we have been trained to essentially ignore)

[fwalch]

@jszakmeister's QuickBuild setup provides some nice charts. For example, on http://neovim-qb.szakmeister.net/statistics/10/junit_report/tests, clicking "Tests" just above the diagram, you can select "compare other metrics" to add "failures" and "errors". You need to switch the target (far right in gray bar above "Tests" caption) to e.g. "functionaltests" and change "group by" to "by version".

[ngortheone]

@justinmk, @blueyed the steps that bot should take are clear, but it is not right to call them "goals". It feels more like a set of steps to solve a problem. Can you elaborate a bit more on "question behind the question" thing?:

• Why such kind of functionality is needed? What problem is solves?
• Is it aimed to improve some existing, but imperfect workflow or solve a new problem that was not yet solved before?
• It feels like a duplication of existing functionality.. Are there problems with current than existing interface that Travis (and other CI systems) provide?
• Who is going to be a primary user for this feature? (you, as nvim org owners, or other contributors)
• Why 20? What is this number related to? Should it be static/hardcoded or configured somehow?
• What is a trigger for "activity"? Comments? Commits? Code review actions?
• Should it also notify gitter (or other chat)?

I have my own guesses and assumptions, but I learned to not relay solely on them as they are often a source of confusion. Anyway, here is my best guess:

The problem:
It takes a lot of mouse clicks to look through build logs, especially if more than one build failed.
A lot of time is wasted to get look through all build logs because of the UIs that travis/appveyor/etc provies.

Solution:
In order to make code reviewers more productive at reviewing PRs (primarily) and as a convenience for contributors (secondarily) an automatic build report should appear in PR conversation section, with build name, status and direct link to raw logs. And it must be updated whenever actual build statuses are updated. (Alternatively only at the end of all CI runs, when status is set to "Checks failed" or "Checks succeeded")

For that to happen the bot must continuously scan PRs for CI activity and add/update comment.

Let me know if my guesses have landed correctly.

[justinmk]

• Why such kind of functionality is needed? What problem is solves?

Viewing build logs is painful, because (1) the CI web UIs are slow, (2) the build logs are big. Goal is to make it less painful. Saving 30-60 seconds, thousands of times, is valuable.

• It feels like a duplication of existing functionality.. Are there problems with current than existing interface that Travis (and other CI systems) provide?

Main functionality is

1. script that creates/updates an automated comment. We don't have that functionality and it is generally useful, regardless of what we put in the automated comment. I don't know of any existing solution. There are various "PR bot" projects but they are complicated and hard to customize.
2. gather build logs. This, again, is generally useful. It's not really important to decide what we'll do with the build logs yet.

• Why 20? What is this number related to? Should it be static/hardcoded or configured somehow?

Hardcoded. It's just an estimate, to avoid spamming GitHub API.

• What is a trigger for "activity"? Comments? Commits? Code review actions?
• Should it also notify gitter (or other chat)?

The script/bot will run e.g. every 30 minutes. I was thinking it could use the GitHub API to sort by "most-recently updated", where "updated" is decided by GitHub.

And it must be updated whenever actual build statuses are updated. (Alternatively only at the end of all CI runs, when status is set to "Checks failed" or "Checks succeeded")

How it's triggered isn't important IMO. We just need a script that uses the GitHub API to gather N most-recently-updated PRs (and their build logs).