v0.14.0 rpts2 cache issue when building `umd` with `cjs`

[AnastasiusVivaldus]

Current Behavior

Right out of the box, running tsdx build --format umd,cjs,esm causes following error:

> tsdx build --format esm,cjs,umd

✓ Creating entry file 1.1 secs
(typescript) Error: ENOENT: no such file or directory, rename '/Users/magnus/work/javascript/myapp/node_modules/.cache/rollup-plugin-typescript2/rpt2_4c61ae4392b9bd24f4d43d13a3f56419b8b5805d/code/cache_' -> '/Users/magnus/work/javascript/myapp/node_modules/.cache/rollup-plugin-typescript2/rpt2_4c61ae4392b9bd24f4d43d13a3f56419b8b5805d/code/cache'
Error: ENOENT: no such file or directory, rename '/Users/magnus/work/javascript/myapp/node_modules/.cache/rollup-plugin-typescript2/rpt2_4c61ae4392b9bd24f4d43d13a3f56419b8b5805d/code/cache_' -> '/Users/magnus/work/javascript/myapp/node_modules/.cache/rollup-plugin-typescript2/rpt2_4c61ae4392b9bd24f4d43d13a3f56419b8b5805d/code/cache'
    at Object.renameSync (fs.js:660:3)
    at RollingCache.roll (/Users/magnus/work/javascript/myapp/node_modules/tsdx/node_modules/rollup-plugin-typescript2/dist/rollup-plugin-typescript2.cjs.js:24003:12)
    at TsCache.done (/Users/magnus/work/javascript/myapp/node_modules/tsdx/node_modules/rollup-plugin-typescript2/dist/rollup-plugin-typescript2.cjs.js:24845:24)
    at Object._ongenerate (/Users/magnus/work/javascript/myapp/node_modules/tsdx/node_modules/rollup-plugin-typescript2/dist/rollup-plugin-typescript2.cjs.js:29098:21)
    at Object.generateBundle (/Users/magnus/work/javascript/myapp/node_modules/tsdx/node_modules/rollup-plugin-typescript2/dist/rollup-plugin-typescript2.cjs.js:29076:18)
    at /Users/magnus/work/javascript/myapp/node_modules/rollup/dist/shared/node-entry.js:13117:25
    at processTicksAndRejections (internal/process/task_queues.js:97:5)

npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! myapp@0.1.0 build: `tsdx build --format esm,cjs,umd`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the myapp@0.1.0 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/magnus/.npm/_logs/2020-10-01T16_45_40_976Z-debug.log

After trying different combinations, I discerned that 'umd' is incompatible with 'cjs'.

Expected behavior

The documentation gives the impression that you can build all three targets simultaneously by running tsdx build --format umd,cjs,esm. If this is not possible to build both 'umd' and 'cjs' types at once then clarify this is in the documentation; otherwise you have a bug here.

Suggested solution(s)

Fix either documentation or underlying problem.

Additional context

Your environment

  System:
    OS: macOS 10.15.6
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Memory: 341.14 MB / 16.00 GB
    Shell: 5.7.1 - /bin/zsh
  Binaries:
    Node: 12.16.1 - ~/.nvm/versions/node/v12.16.1/bin/node
    Yarn: 1.22.5 - ~/.nvm/versions/node/v12.16.1/bin/yarn
    npm: 6.13.4 - ~/.nvm/versions/node/v12.16.1/bin/npm
  Browsers:
    Brave Browser: 85.1.14.81
    Chrome: 85.0.4183.121
    Firefox: 81.0
    Safari: 14.0
  npmPackages:
    tsdx: ^0.14.0 => 0.14.0
    typescript: ^4.0.3 => 4.0.3

[agilgur5]

Based on the logs, sounds like a duplicate of #888. (EDIT: I've now marked that issue as duplicative of this issue because this one has more details now.) Per my response there:

Looks like this is a cache bug upstream that was recently fixed: ezolenko/rollup-plugin-typescript2#243

Per comments there, you can workaround this by deleting the cache. I don't think you need to fully nuke node_modules, think you can just delete node_modules/.cache/rollup-plugin-typescript2/

After trying different combinations, I discerned that 'umd' is incompatible with 'cjs'.

That's an interesting find though upstream in rollup-plugin-typescript2 it has no knowledge of format (I've worked on that codebase several times now), so more likely it's some bad hash and race condition that happens during updating

The documentation gives the impression that you can build all three targets simultaneously

You can, a central piece of TSDX and even Rollup itself, and there are tests for this too. You just appear to have hit a transient cache bug upstream, which the logs point to.

[agilgur5]

So I tried a few things and confirmed the umd + cjs issue. I also confirmed it was happening with system + cjs. But not esm + cjs for some reason. I also tried ordering it differently, but that didn't change anything (they are parallelized in any case but 🤷 ). I also confirmed umd + system had the issue as well.

The system addition is interesting, as that shouldn't have much difference from the others aside from format. system + umd also knocks the suggestion that it's some cjs issue out 😕 😕

Here's some logs:

$$ tsdx create test-tsdx && cd test-tsdx
[...]

$$ yarn build --format umd
yarn run v1.22.4
$ tsdx build --format umd
✓ Building modules 674 ms
✨  Done in 3.96s.

$$ yarn build --format cjs,umd
yarn run v1.22.4
$ tsdx build --format cjs,umd
✓ Creating entry file 967 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format system,umd
yarn run v1.22.4
$ tsdx build --format system,umd
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format cjs,esm
yarn run v1.22.4
$ tsdx build --format cjs,esm
✓ Creating entry file 957 ms
✓ Building modules 1.1 secs
✨  Done in 3.86s.

$$ yarn build --format cjs,esm,umd
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd
✓ Creating entry file 954 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format cjs,esm,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,system
✓ Creating entry file 963 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format cjs,system
yarn run v1.22.4
$ tsdx build --format cjs,system
✓ Creating entry file 969 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'

$$ yarn build --format esm,system
yarn run v1.22.4
$ tsdx build --format esm,system
✓ Building modules 1.2 secs
✨  Done in 3.94s.

$$ yarn build --format esm,umd
yarn run v1.22.4
$ tsdx build --format esm,umd
✓ Building modules 634 ms
✨  Done in 3.36s.

# different ordering
$$ yarn build --format esm,cjs,umd
yarn run v1.22.4
$ tsdx build --format esm,cjs,umd
✓ Creating entry file 980 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

# different ordering
$$ yarn build --format esm,umd,cjs
yarn run v1.22.4
$ tsdx build --format esm,umd,cjs
✓ Creating entry file 979 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format esm,umd,system
yarn run v1.22.4
$ tsdx build --format esm,umd,system
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_dc5aaa4ea403c2ba50e68d0baecc90e56a88ad12/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_dc5aaa4ea403c2ba50e68d0baecc90e56a88ad12/code/cache'
[...]

$$ yarn build --format umd,system
yarn run v1.22.4
$ tsdx build --format umd,system
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'
[...]

$$ yarn build --format umd,cjs
yarn run v1.22.4
$ tsdx build --format umd,cjs
✓ Creating entry file 962 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/oss/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache_' -
> '[redacted]/oss/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_d8917360bd7c6c37111985035e75a4930df504e4/code/cache'

[agilgur5]

Additionally confirmed that this issue did exist in TSDX v0.13.3, but only for the cjs + system combo, which is much less frequent. Was confused that it hadn't happened with v0.13.3 despite it using the same version of rpts2 as v0.14.0, but this allays some of that. And also makes it seem more of a hashing bug or race condition upstream than a specific format, which would've been confusing too given the lack of knowledge of format.

More logs:

$ yarn add tsdx@0.13.3
yarn add v1.22.4
[...]
$ tsdx build
✓ Creating entry file 1.3 secs
✓ Building modules 2.2 secs
✨  Done in 17.88s.

$$ yarn build --format umd,cjs
yarn run v1.22.4
$ tsdx build --format umd,cjs
✓ Creating entry file 1 secs
✓ Building modules 1.9 secs
✨  Done in 4.10s.

$$ yarn build --format umd,cjs,esm,system
yarn run v1.22.4
$ tsdx build --format umd,cjs,esm,system
✓ Creating entry file 919 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache'
[...]

$$ yarn build --format umd,esm
yarn run v1.22.4
$ tsdx build --format umd,esm
✓ Building modules 715 ms
✨  Done in 2.68s.

$$ yarn build --format umd,esm,cjs
yarn run v1.22.4
$ tsdx build --format umd,esm,cjs
✓ Creating entry file 924 ms
✓ Building modules 1.4 secs
✨  Done in 3.40s.

$$ yarn build --format umd,system
yarn run v1.22.4
$ tsdx build --format umd,system
✓ Building modules 907 ms
✨  Done in 2.86s.
✔ ~/Desktop/GitHub/oss/test-tsdx (⎈ |services.k8s.dev.ns8.ninja:default) 

$$ yarn build --format cjs,system
yarn run v1.22.4
$ tsdx build --format cjs,system
✓ Creating entry file 915 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache'
[...]

$$ yarn build --format cjs,esm,umd
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd
✓ Creating entry file 924 ms
✓ Building modules 1.4 secs
✨  Done in 3.43s.

$$ yarn build --format umd,system
yarn run v1.22.4
$ tsdx build --format umd,system
✓ Building modules 897 ms
✨  Done in 2.87s.

$$ yarn build --format cjs,system
yarn run v1.22.4
$ tsdx build --format cjs,system
✓ Creating entry file 913 ms
(typescript) Error: ENOENT: no such file or directory, rename '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache_' -
> '[redacted]/test-tsdx/node_modules/.cache/rollup-plugin-typescript2/rpt2_57678ef6f24a79cbc0a37de5d02872e29b66702a/code/cache'
[...]

$$ yarn build --format cjs,esm,umd
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd
✓ Creating entry file 914 ms
✓ Building modules 1.4 secs
✨  Done in 3.34s.

[agilgur5]

Hmmm v0.13.2, v0.13.1, v0.13.0, and v0.12 don't have this issue 🤔 That being said, I'm not sure I'd call this a regression per se, because, per my investigation in #896 and ezolenko/rollup-plugin-typescript2#243, that section of code upstream has been like that for a long time. But TSDX's cache directories and Rollup config (which impacts the hash) have changed over time. 🤔

Side note: the perf decrease when downgrading was noticeable, so that confirms the more recent optimizations I've made help in a variety of codebase sizes 🙂

v0.13.3 consolidated the cache directories for performance reasons (more matching hashes); previously there was a different one for each format, so perhaps that creates more races or makes for an incorrectly matched hash. The problematic cache seems to have an identical hash in all of the logs (only one is different from the rest) 🤔

Logs:

$$ yarn add tsdx@0.13.2
yarn add v1.22.4
[...]
$ tsdx build
✓ Creating entry file 1.8 secs
✓ Building modules 753 ms
✨  Done in 7.66s.

$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.4 secs
✓ Building modules 1.3 secs
✨  Done in 5.88s.

$$ yarn build --format umd,system
yarn run v1.22.4
$ tsdx build --format umd,system
✓ Building modules 944 ms
✨  Done in 4.15s.

$$ yarn build --format umd,cjs,esm
yarn run v1.22.4
$ tsdx build --format umd,cjs,esm
✓ Creating entry file 2.6 secs
✓ Building modules 912 ms
✨  Done in 4.65s.

$$ yarn build --format umd,cjs
yarn run v1.22.4
$ tsdx build --format umd,cjs
✓ Creating entry file 2.1 secs
✓ Building modules 985 ms
✨  Done in 4.18s.

$$ yarn build --format umd,esm
yarn run v1.22.4
$ tsdx build --format umd,esm
✓ Building modules 568 ms
✨  Done in 3.46s.

$$ yarn build --format cjs,system
yarn run v1.22.4
$ tsdx build --format cjs,system
✓ Creating entry file 2.2 secs
✓ Building modules 939 ms
✨  Done in 4.22s.

$$ yarn build --format esm,system
yarn run v1.22.4
$ tsdx build --format esm,system
✓ Building modules 618 ms
✨  Done in 3.43s.

$$ yarn build --format esm,umd,system
yarn run v1.22.4
$ tsdx build --format esm,umd,system
✓ Building modules 1.1 secs
✨  Done in 4.93s.

$$ yarn build --format cjs,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,umd,system
✓ Creating entry file 3 secs
✓ Building modules 1.2 secs
✨  Done in 5.25s.

$$ yarn build --format cjs,esm,umd
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd
✓ Creating entry file 2.6 secs
✓ Building modules 916 ms
✨  Done in 4.63s.

$$ yarn add tsdx@0.13.1
yarn add v1.22.4
[...]
$ tsdx build
✓ Creating entry file 1.8 secs
✓ Building modules 686 ms
✨  Done in 8.09s.

$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.5 secs
✓ Building modules 1.3 secs
✨  Done in 5.91s.

# second time for cache etc
$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.5 secs
✓ Building modules 1.3 secs
✨  Done in 5.87s.

$$ yarn add tsdx@0.13.0
yarn add v1.22.4
[...]
$ tsdx build
✓ Creating entry file 1.9 secs
✓ Building modules 652 ms
✨  Done in 7.44s.

$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.4 secs
✓ Building modules 1.2 secs
✨  Done in 5.75s.

# second time for cache etc
$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.5 secs
✓ Building modules 1.2 secs
✨  Done in 5.76s.

$$ yarn add tsdx@0.12
[...]
$ tsdx build
✓ Creating entry file 2.4 secs
✓ Building modules 960 ms
✨  Done in 9.50s.

$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.7 secs
✓ Building modules 1.4 secs
✨  Done in 6.28s.

# second time for cache etc
$$ yarn build --format cjs,esm,umd,system
yarn run v1.22.4
$ tsdx build --format cjs,esm,umd,system
✓ Creating entry file 3.7 secs
✓ Building modules 1.3 secs
✨  Done in 6.14s.