beta testing github-app

[lukpueh]

Here are some observations from testing the github-app on a test repo following the getting started guide.

• For some reason the installation docs made me think that there would be any gittuf specific configuration steps during the install, Maybe because of sentence "The UI will walk you through the standard installation process. As a part of this..." . So I was surprised, that the installation was finished after I chose an account to install, and briefly thought the installation had failed.
• Removing the quoted phrase might make the instruction clearer.
• After the installation I was redirected to the top of the getting started document I was coming from, and already two thirds through reading.
• The redirect should lead somewhere, where I should go next after the installation.
• I then created a PR and noticed the gittuf-app check "PR is not mergeable" (and in the details: "More approvals are necessary for the PR to be mergeable.")
• Where does this policy come from?
• Is it hardcoded into the app?
• I couldn't find docs, or settings for this.
• Does the app install a gittuf policy into the git metadata?
• How many approvals are actually needed?
• Also gittuf and GitHub give me mixed signals about the mergeability of my PR.
  • gittuf: Verify gittuf policy / PR is not mergable
  • GitHub: Merging can be performed automatically / Merge pull request
• Despite gittuf saying otherwise, merging seems to work
• How can I see the merge attestation now?
• When I fetch locally after merge, I do find an RSL Reference Entry (requires git/gittuf specific knowledge)

   git fetch origin "refs/gittuf/*:refs/gittuf/*" && git cat-file -p $(cat .git/refs/gittuf/reference-state-log)
  

• Are there no attestations because I didn't approve the PR?
• Second round: PR with approval
• One approval does not seem to be enough. gittuf still says "More approvals are necessary for the PR to be mergeable."
• GitHub says otherwise
• Merging and fetching locally, I still don't see any attestations. But refs/gittuf/reference-state-log points to the second RSL Reference Entry now.
• Maybe much of this can be solved with better documentation / user feedback. :)

[adityasaky]

I then created a lukpueh/test-gittuf-github-app#1 and noticed the gittuf-app check "PR is not mergeable" (and in the details: "More approvals are necessary for the PR to be mergeable.")
Where does this policy come from?
Is it hardcoded into the app?
I couldn't find docs, or settings for this.
Does the app install a gittuf policy into the git metadata?
How many approvals are actually needed?

Thanks for flagging this, agreed it's confusing. This should be fixed as of #123. As an explanation, what happened was that the gittuf app tried to verify the policy to see if the PR is mergeable. In "lite" mode, there is no policy, so the verify-mergeable workflow failed. In turn, the app indicated there were insufficient approvals. When we added that error message to the app, the only reason for verify-mergeable to fail was if there were insufficient approvals, which is clearly incompatible with lite mode. We've updated the app to skip the checkrun entirely if the repository has no policy.

Does the app install a gittuf policy into the git metadata?

Nope!

[adityasaky]

Also gittuf and GitHub give me mixed signals about the mergeability of my PR.

• gittuf: Verify gittuf policy / PR is not mergable
• GitHub: Merging can be performed automatically / Merge pull request

So, in lite mode, the first issue ought to be fixed, but you raise a bigger question about inconsistencies between a repo's gittuf policy and the configured github rulesets. Something may be mergeable according to the github rule but not be mergeable per the gittuf policy, and vice versa. This should be documented for the full mode experience.

[adityasaky]

Also, note that the RSL's tree being empty is expected. The attestations are stored in refs/gittuf/attestations. I just opened a test PR on that repo and you can see the RSL entry for the approval here https://github.com/lukpueh/test-gittuf-github-app/commits/gittuf/reference-state-log. You'll see the latest entry lukpueh/test-gittuf-github-app@2dd65dc points to refs/gittuf/attestations.

Your earlier attempts seem to have hit...an ephemeral issue 😓, which was why there wasn't an attestation created. Working on debugging that!