Enable un-attended bootstrapping for `pk agent start` and `pk bootstrap` with `PK_RECOVERY_CODE` and `PK_PASSWORD`

[CMCDragonkai]

Block 1 from #194 (comment)

Aims to provide support for a Polykey instance to run unattended (i.e. without user prompt) when both the PK_RECOVERY_CODE and PK_PASSWORD variables are set.

Requirements of this design

1. User has to supply a root password in order to encrypt a root key that is automatically generated
2. If the user loses or forgets the root password, there is no way to decrypt the root key, there is no "Forget Password" functionality for decentralised application
3. Provider an alternative way of recovering the root key
4. The alternative is to store a BIP39 seed phrase that is used to generate the root key. If the user can supply a BIP39 seed phrase, this can be used to deterministically generate the root key, which doesn't require decrypting the root key and thus unlocking the keynode
5. In an extreme case, the root key ciphertext file on disk is corrupted, and so decrypting it produces an incorrect root key, the seed phrase represents a backup of the root key
6. Users must be notified to securely backup/store their root key, offer users to print it out as a "paper key" (like https://www.jabberwocky.com/software/paperkey/, we can even do funny things like QR codes)

Additional context

• https://gitlab.com/MatrixAI/Engineering/Polykey/Polykey/-/issues/75
• Deterministic generation of the pki.rsa.generateKeyPair digitalbazaar/forge#865

Note that the core function to generate a root key from a BIP39 seed is already available in our keys/utils.ts, see the function: generateDeterministicKeyPair. However it is not currently used by KeyManager, and not well tested.

Screenshot from Figma GUI design:

Specification

1. Get the KeyManager to use the generateDeterministicKeyPair
2. Make sure that this function generateDeterministicKeyPair is well tested
3. Users are presented with a 24 word phrase to store and remember (prompt the user to store/print it)
4. During the startup of the PK agent, the user has to supply the root password OR the seed phrase
5. During session unlock, the user can supply the root password OR the seed phrase
6. When a seed phrase is supplied, the root key is regenerated from the seed phrase, the root key ciphertext is not touched
7. We must have a root key acceptance condition to know whether the root key has legitmately opened up a vault
8. If it has, we can save/rewrite the root key as ciphertext over the ciphertext in the node state, which will resolve any root key ciphertext corruption

Regarding the root key acceptance, consider this idea:

• Create a signed-message
• Encrypt the message and save the message
• If you can later decrypt the message, and check the the signature was signed by the root key and the message is what you expect
• Then the root key is correct!
• This message will need to be stored in the DB (as this is what is decrypted by the root key)

CLI command variables

Variables for CLI commands can be retrieved from:

• file parameters
• environment variables
• src/config.ts
• user prompt in STDIN

We require functions to retrieve the variables from each of these locations (created in src/bin/utils).

Variable precedence should be as follows: file parameters > environment variables/src/config.ts > default (STDIN prompt). We can use a sequence of coalescing operations to emulate this: for example, to retrieve the password, password = getFromParams() ?? getFromEnv ?? getFromStdin(). getFromStdin() here is expected to be the default, where we assume that a value will always be provided. If a default cannot exist for a particular variable, then throw an exception if none of the sources of variables can produce one.

See point 1 at #202 (comment) for further information.

Bootstrap process

Some components of the bootstrapping process need to be simplified:

• bootstrapPolykeyState (see 3 Enable un-attended bootstrapping for pk agent start and pk bootstrap with PK_RECOVERY_CODE and PK_PASSWORD #202 (comment))
  • ensure that the lockfile is created in the appropriate way (see writeToken in Session.ts - discuss with @CMCDragonkai
  • then, for checking whether there's an existing agent process, do the following:
    1. If the lock file exists:
    2. Attempt to acquire the lock
    3. If we cannot acquire the lock, then something is running. If this is the case, we can simply abort the process. No need to await until the lock is released.
  • i.e. no PID checks required
• checkKeynodeState (see 4 Enable un-attended bootstrapping for pk agent start and pk bootstrap with PK_RECOVERY_CODE and PK_PASSWORD #202 (comment))
  • make this more coarse (less fine-grained) on the contents of the nodePath: if any state exists besides the lockfile, throw an exception

Sub-Issues & Sub-PRs created

#283

checklist

• Generate the keypair from the BIP menmonic
  • Generate a BIP mnemonic
  • generate keypair from menmonic
• update keyManager to use new key generation method.
• add a check that the mnemonic matches the nodeId for the node. if so we update the encrypted rootkey with the new password.
• fix KeyManager to do the recovery process when a recovery code is provided with a password when node state exists.
• tests
  • generating a keypair.
  • mnemonic deterministically recreates keypair.
  • can tell if mnemonic matches keypair/nodeId
  • can create new node from scratch, recreates nodeId and keypair.
  • can recover a keynode.
• env variables implementation
  • bin/utils functions for variable retrieval
    • ~~getFromParams(): get variable from file parameter. Use str.trim() to remove trailing newlines/whitespace from the file (see 2 at Enable un-attended bootstrapping for pk agent start and pk bootstrap with PK_RECOVERY_CODE and PK_PASSWORD #202 (comment) for further information) ~~
    • getFromEnv(): get variable from environment variable
    • getFromConfig(): get variable from src/config.ts
    • getFromStdin(): get variable from user prompt
    • changed how this is handled, there are binUtils.getPassword and binUtils.getRecoveryCode. options for hosts and ports was created in src/bin/options.ts with their own defaults and env.
• bootstrap process
  • use ENV variables for password and recovery code.
  • simplify bootstrapPolykeyState
  • tests for:
    • concurrent execution of 2x pk bootstrap
    • concurrent execution of 2x pk agent start
    • concurrent execution of pk bootstrap + pk agent start
  • simplify checkKeynodeState
• implement --fresh flag: forces new keynode state, even if it already exists
  • pk agent start
  • pk bootstrap

[CMCDragonkai]

The BIP39 spec is here: https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki

The implementation of the function is currently:

async function generateDeterministicKeyPair(
  bits: number,
  seed: string,
): Promise<KeyPair> {
  const prng = random.createInstance();
  prng.seedFileSync = (needed: number) => {
    // using bip39 seed generation parameters
    // no passphrase is considered here
    return pkcs5.pbkdf2(seed, 'mnemonic', 2048, needed, md.sha512.create());
  };
  const generateKeyPair = promisify(pki.rsa.generateKeyPair).bind(pki.rsa);
  return await generateKeyPair({ bits, prng });
}

The seed is actually called the "mnemonic phrase". This is meant to be generated by a bip39 library: https://www.npmjs.com/package/bip39

The "mnemonic phrase" is converted to a binary seed using PBKDF2. This binary seed is what is used to generate the RSA root keypair. As long as the same seed can be generated from the same mnemonic phrase, then the same keypair will come out.

Notice that the salt is currently just mnemonic.

In the BIP39 spec, the user can specify a passphrase to protect the the mnemonic seed. To do so, they must provide a "passphrase" that is appended to the end of 'mnemonic' + passphrase used for the salt. This usecase doesn't make sense for Polykey.

We are already randomly generating the root keypair. Switching over to BIP39 simply means we use:

1. bip39 library to randomly generate a mnemonic phrase of 12/24 words
2. generateDeterministicKeyPair to generate the root key pair
3. The root keypair is saved to the node state, while requesting the user a password to symmetrically encrypt it, this is using keyPairToPemEncrypted, which uses a special encryptRsaPrivateKey function from pki, this has its own spec.
4. Thus there's no need for a separate passphrase for adding to the salt of bip39.
5. Finally you have a root password that can decrypt the ciphered root keypair, and if you lose the root keypair or forget your root password, you can recover the plaintext root keypair by using generateDeterministicKeyPair again with your 12/24 word mnemonic phrase.

[CMCDragonkai]

Some options here for the pk bootstrap command:

• Should PK_PASSWORD which is used to supply password for authentication of PK cli commands be used by pk bootstrap to also automatically use as the password without prompting?
• If the PK_RECOVERY_CODE is supplied, this forces a deterministic creation of the root key. Then the root key can be protected by PK_PASSWORD if supplied, otherwise there is a prompt for PK_PASSWORD

The semantics of these environment variables would be specific to the pk bootstrap command, and therefore should be centralised there.

To ensure un-attended pk bootstrap and pk agent start all parameters must be supplied either by env variables or by parameters. If supplied with parameters, then the parameters takes over from the env variables.

[tegefaulkes]

SHould the Bip recovery function separate from the KeyManager or a static method? We don't really need to start the KeyManager to do the recovery but we might need use the DB depending how I handle checking if the Mnemonic is for the node.

[CMCDragonkai]

I believe that the deterministic key generation should be done all the time.

That means when calling pk bootstrap, if there's no PK_RECOVERY_CODE available, that means you will be generating one.

When pk bootstrap command finishes, it should have:

1. generated random recovery code
2. used recovery code to generate root key
3. prompted you for the root password to encrypt the root key
4. present the user the recovery code on the STDOUT

This means recovery code is always generated.

[CMCDragonkai]

I've renamed this issue to better represent the intention. The usage of BIP39 is an implementation detail of the PK_RECOVERY_CODE, and we have to make sure this works for both pk agent start and pk bootstrap.

Recent discussions with @tegefaulkes indicates that pk agent start automatically bootstraps as well. So pk bootstrap is only used when you want to bootstrap without starting an agent. But if you are starting an agent you can bootstrap automatically.

That means pk agent start will not bootstrap when it detects the directory is already occupied. But we must deal with the case where the directory is occupied by not a proper node state, any errors during the start will result in failure to start. A fresh pk agent start would delete the directory, but but this must be explicit like we do in the code with the fresh boolean.

Same behaviour then with pk bootstrap too.