diff --git a/AGENTS.md b/AGENTS.md index 825897e..2f62ab8 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -23,6 +23,18 @@ Check if services are running: `bash dev/status.sh` Log files are in `dev/logs/` (dev-api.log, dev-ui.log, docker-compose.log). +### Vulnerability scanner (osv-scanner) + +Dev config enables npm vulnerability scanning (`api/config/development.js`), which shells out to the **osv-scanner v2** binary. Install it once so it's on your PATH (Linux x86_64): + + OSV_VERSION=v2.2.3 + sudo curl -sSfL -o /usr/local/bin/osv-scanner \ + "https://github.com/google/osv-scanner/releases/download/${OSV_VERSION}/osv-scanner_linux_amd64" + sudo chmod +x /usr/local/bin/osv-scanner + osv-scanner --version + +(Any directory on PATH works; drop `sudo` if you install into a user-writable dir.) The offline OSV database (~200 MB) downloads on first scan into `./data/osv-db` and is refreshed by the periodic rescan; extracted artefacts are cached under `./data/tmp/scan-cache`. If osv-scanner is not installed, scans fail gracefully (`scan.status: "error"`) and the rest of the app keeps working. + ## Testing Tests use Playwright as a test runner with 3 project types: unit (*.unit.spec.ts), api (*.api.spec.ts), and e2e (*.e2e.spec.ts). State setup/teardown fixtures run before api and e2e tests. @@ -37,6 +49,18 @@ Run a specific test file: Test users are defined in @dev/resources/users.json and organizations in @dev/resources/organizations.json. +## Vulnerability scanning + +Advisory, admin-only vulnerability scanning of `npm` artefacts (bundled `node_modules` tarballs) via the bundled **osv-scanner v2** binary. It never blocks uploads or downloads, and scan data is stripped from responses for non-admin callers. + +- Controlled by the `scanning.*` config block (`api/config/type/schema.json`); **off by default** (`scanning.enabled`). +- osv-scanner runs in **offline local-DB mode** (`scanning.dbDir`), refreshed on the rescan interval. Bundled, lockfile-less `node_modules` are detected via the `--experimental-plugins javascript/packagejson` extractor on `osv-scanner scan source`. +- Scans run on three triggers: after an npm upload (async, non-blocking), on a periodic interval (`scanning.rescanIntervalHours`, also refreshes the DB), and on-demand via `POST /api/v1/artefacts/:id/scan` (admin). +- A summary lives on the artefact `scan` field (admin-only); full findings are stored in the `artefact-scans` Mongo collection and served by `GET /api/v1/artefacts/:id/scan` (admin). +- Module: `api/src/scanning/` (`operations.ts` pure mapping, `extract.ts` tarball extraction, `runner.ts` osv-scanner subprocess, `service.ts` orchestration, `router.ts` endpoints). +- The osv-scanner binary is bundled in the Docker image (see `Dockerfile`). The mapper's test fixture is `tests/resources/osv-sample-output.json`. +- Extracted artefacts are cached as a mirror under `/scan-cache/` (config `tmpDir`, env `TMP_DIR`; mount as a k8s `emptyDir`). A scan reuses the cached extraction when the artefact's bytes are unchanged (keyed on `artefact.path`); `rescanAll` prunes slots for deleted artefacts. + ## Code patterns - API route: @api/src/artefacts/router.ts diff --git a/Dockerfile b/Dockerfile index dbc064c..49bdaad 100644 --- a/Dockerfile +++ b/Dockerfile @@ -17,6 +17,17 @@ ADD package.json package-lock.json ./ RUN jq '.version="build"' package.json | sponge package.json RUN jq '.version="build"' package-lock.json | sponge package-lock.json +########################## +FROM base AS osv-scanner +# Advisory npm vulnerability scanner (statically-linked Go binary; runs on musl). +ARG OSV_SCANNER_VERSION=v2.2.3 +ARG TARGETARCH=amd64 +RUN apk add --no-cache curl && \ + curl -sSfL -o /usr/local/bin/osv-scanner \ + "https://github.com/google/osv-scanner/releases/download/${OSV_SCANNER_VERSION}/osv-scanner_linux_${TARGETARCH}" && \ + chmod +x /usr/local/bin/osv-scanner && \ + /usr/local/bin/osv-scanner --version + ########################## FROM base AS installer @@ -77,6 +88,7 @@ RUN mkdir -p /app/api/node_modules FROM native-deps AS main COPY --from=api-installer /app/node_modules node_modules +COPY --from=osv-scanner /usr/local/bin/osv-scanner /usr/local/bin/osv-scanner ADD /api api ADD /shared shared COPY --from=types /app/api/types api/types diff --git a/api/config/custom-environment-variables.js b/api/config/custom-environment-variables.js index 7d7a629..aec0f39 100644 --- a/api/config/custom-environment-variables.js +++ b/api/config/custom-environment-variables.js @@ -15,6 +15,7 @@ export default { }, upgradeRoot: 'UPGRADE_ROOT', dataDir: 'DATA_DIR', + tmpDir: 'TMP_DIR', maxUploadBytes: { __name: 'MAX_UPLOAD_BYTES', __format: 'number' @@ -27,6 +28,26 @@ export default { __name: 'MAX_TAR_ENTRIES', __format: 'number' }, + scanning: { + enabled: { + __name: 'SCANNING_ENABLED', + __format: 'boolean' + }, + osvScannerPath: 'OSV_SCANNER_PATH', + dbDir: 'OSV_DB_DIR', + concurrency: { + __name: 'SCANNING_CONCURRENCY', + __format: 'number' + }, + rescanIntervalHours: { + __name: 'SCANNING_RESCAN_INTERVAL_HOURS', + __format: 'number' + }, + timeoutSeconds: { + __name: 'SCANNING_TIMEOUT_SECONDS', + __format: 'number' + } + }, filesStorage: 'FILES_STORAGE', s3: { region: 'S3_REGION', diff --git a/api/config/default.js b/api/config/default.js index 98747b7..e0a3f1c 100644 --- a/api/config/default.js +++ b/api/config/default.js @@ -4,9 +4,18 @@ export default { mongoUrl: 'mongodb://localhost:27017/data-fair-registry', port: 8080, dataDir: '/data', + tmpDir: undefined, maxUploadBytes: 200 * 1024 * 1024 * 1024, maxDecompressedBytes: 1024 * 1024 * 1024, maxTarEntries: 100_000, + scanning: { + enabled: false, + osvScannerPath: 'osv-scanner', + dbDir: '/data/osv-db', + concurrency: 1, + rescanIntervalHours: 24, + timeoutSeconds: 300 + }, filesStorage: 'fs', s3: { region: '', diff --git a/api/config/development.js b/api/config/development.js index b86e904..301c545 100644 --- a/api/config/development.js +++ b/api/config/development.js @@ -9,6 +9,15 @@ export default { privateEventsUrl: `http://localhost:${process.env.EVENTS_PORT}`, mongoUrl: `mongodb://localhost:${process.env.MONGO_PORT}/data-fair-registry-development`, dataDir: './data', + // Vulnerability scanning is enabled in dev. Requires the osv-scanner binary + // on PATH (see "Vulnerability scanner" in AGENTS.md for install). The offline + // OSV DB is downloaded under dataDir on first scan. If osv-scanner is missing, + // scans just record scan.status="error" and the rest of the app is unaffected. + scanning: { + enabled: true, + osvScannerPath: 'osv-scanner', + dbDir: './data/osv-db' + }, observer: { active: false }, diff --git a/api/config/type/schema.json b/api/config/type/schema.json index df6eeba..d903b09 100644 --- a/api/config/type/schema.json +++ b/api/config/type/schema.json @@ -27,6 +27,10 @@ "dataDir": { "type": "string" }, + "tmpDir": { + "type": "string", + "description": "Working directory for ephemeral scan extractions (the scan cache lives at /scan-cache). Defaults to /tmp, or an OS temp dir. Mount as an emptyDir in k8s." + }, "maxUploadBytes": { "type": "integer", "default": 524288000, @@ -42,6 +46,46 @@ "default": 100000, "description": "Maximum number of entries (files + directories) accepted in an uploaded npm tarball. Pre-installed node_modules can easily exceed the lower legacy default." }, + "scanning": { + "type": "object", + "additionalProperties": false, + "default": {}, + "properties": { + "enabled": { + "type": "boolean", + "default": false, + "description": "Enable advisory vulnerability scanning of npm artefacts via osv-scanner. Off by default so deployments without the binary keep working." + }, + "osvScannerPath": { + "type": "string", + "default": "osv-scanner", + "description": "Path or command name of the osv-scanner v2 binary." + }, + "dbDir": { + "type": "string", + "default": "/data/osv-db", + "description": "Local directory holding the offline OSV vulnerability database." + }, + "concurrency": { + "type": "integer", + "minimum": 1, + "default": 1, + "description": "Maximum number of artefact scans running at once in this process." + }, + "rescanIntervalHours": { + "type": "integer", + "minimum": 1, + "default": 24, + "description": "Interval in hours between automatic rescans of all npm artefacts (also refreshes the OSV DB)." + }, + "timeoutSeconds": { + "type": "integer", + "minimum": 30, + "default": 300, + "description": "Per-artefact scan timeout in seconds (extraction + osv-scanner)." + } + } + }, "privateDirectoryUrl": { "type": "string", "pattern": "^https?://" diff --git a/api/package.json b/api/package.json index 2dd1d64..c996279 100644 --- a/api/package.json +++ b/api/package.json @@ -34,9 +34,10 @@ "mongodb": "^6.8.0", "piscina": "^4.7.0", "prom-client": "^15.1.3", - "sharp": "^0.33.5", "resolve-path": "^1.4.0", "semver": "^7.6.0", + "sharp": "^0.33.5", + "tar": "^7.5.16", "tar-stream": "^3.1.0" } } diff --git a/api/src/app.ts b/api/src/app.ts index d08afdc..8282d87 100644 --- a/api/src/app.ts +++ b/api/src/app.ts @@ -44,6 +44,7 @@ if (process.env.NODE_ENV === 'development') { app.delete('/api/test-env', async (req, res) => { assertReqInternal(req) await mongo.artefacts.deleteMany({}) + await mongo.artefactScans.deleteMany({}) await mongo.apiKeys.deleteMany({}) // Scoped delete: only grants for test accounts (test users/orgs are // `test*`-prefixed) are wiped, so manually-created dev grants survive. @@ -64,6 +65,47 @@ if (process.env.NODE_ENV === 'development') { } res.send() }) + + app.put('/api/test-env/artefacts/:id/scan', async (req, res) => { + assertReqInternal(req) + const id = decodeURIComponent(req.params.id) + const { summary, findings } = req.body + await mongo.artefacts.updateOne( + { _id: id }, + { $set: { scan: { status: 'success', finishedAt: new Date().toISOString(), summary } } } + ) + await mongo.artefactScans.replaceOne( + { _id: id }, + { scannedAt: new Date().toISOString(), scannerVersion: 'test', vulnerabilities: findings ?? [] }, + { upsert: true } + ) + res.send() + }) + + // Upsert an artefact doc directly with arbitrary fields — lets tests build a + // deterministic fleet (with controlled scan state) without the upload + // pipeline, whose auto-scan would otherwise race injected state. + app.put('/api/test-env/artefacts/:id/doc', async (req, res) => { + assertReqInternal(req) + const id = decodeURIComponent(req.params.id) + const now = new Date().toISOString() + await mongo.artefacts.replaceOne( + { _id: id }, + { + _id: id, + name: id, + format: 'npm', + public: false, + privateAccess: [], + createdAt: now, + updatedAt: now, + dataUpdatedAt: now, + ...req.body + } as any, + { upsert: true } + ) + res.send() + }) } app.use('/api', (req, res) => res.status(404).send('unknown api endpoint')) diff --git a/api/src/artefacts/router.ts b/api/src/artefacts/router.ts index d3decd3..8c91ec7 100644 --- a/api/src/artefacts/router.ts +++ b/api/src/artefacts/router.ts @@ -14,15 +14,29 @@ import { filesStorage } from '../files-storage/index.ts' import { listArtefacts, getArtefact, getArtefactById, patchArtefact, deleteArtefact, commitFileUpload, commitNpmUpload, extractStagedManifest, resolveDownload, - listGroupValues + listGroupValues, getScanSummary } from './service.ts' import * as patchReqBody from '#doc/artefacts/patch-req/index.ts' import { artefactThumbnailRouter } from '../thumbnails/router.ts' +import scanRouter from '../scanning/router.ts' +import type { Caller } from '../access.ts' const router = Router() export default router router.use('/:id/thumbnail', artefactThumbnailRouter) +router.use('/:id/scan', scanRouter) + +// Drop the advisory, admin-only scan field unconditionally. +const omitScan = (artefact: T): T => { + if (artefact.scan === undefined) return artefact + const { scan, ...rest } = artefact + return rest as T +} + +// Scan results are advisory and admin-only; strip them for non-admin callers. +const stripScan = (artefact: T, caller: Caller): T => + caller.admin ? artefact : omitScan(artefact) const npmCategories = ['processing', 'catalog', 'application', 'other'] as const const fileCategories = ['tileset', 'maplibre-style', 'other'] as const @@ -85,10 +99,18 @@ const tryInternalSecret = (req: import('express').Request): boolean => { // List artefacts (filtered by access) router.get('/', async (req, res, next) => { try { - const filter = artefactAccessFilter(await resolveCaller(req)) + const caller = await resolveCaller(req) + const filter = artefactAccessFilter(caller) const skip = Math.max(0, Math.min(parseInt(req.query.skip as string) || 0, 100000)) const size = Math.min(parseInt(req.query.size as string) || 10, 100) - const sort: Record = req.query.sort === 'name' ? { name: 1 } : { dataUpdatedAt: -1 } + // `vulnerabilities` is admin-only: scan data is stripped for non-admins, so + // ordering by it would leak. Non-admins silently get the default sort. + let sort: Record = { dataUpdatedAt: -1 } + if (req.query.sort === 'name') { + sort = { name: 1 } + } else if (req.query.sort === 'vulnerabilities' && caller.admin) { + sort = { 'scan.summary.critical': -1, 'scan.summary.high': -1, 'scan.summary.medium': -1, dataUpdatedAt: -1 } + } // Text search on name if (req.query.q) { @@ -116,7 +138,7 @@ router.get('/', async (req, res, next) => { } const { results, count } = await listArtefacts(filter, { sort, skip, size }) - res.json({ results, count }) + res.json({ results: results.map(r => stripScan(r, caller)), count }) } catch (err) { next(err) } }) @@ -136,14 +158,23 @@ router.get('/groups', async (req, res, next) => { } catch (err) { next(err) } }) +// Fleet-wide vulnerability roll-up for the admin dashboard. Registered before +// GET /:id so the literal path isn't swallowed as an id. Admin-only. +router.get('/scan-summary', async (req, res, next) => { + try { + await session.reqAdminMode(req) + res.json(await getScanSummary()) + } catch (err) { next(err) } +}) + // Get artefact detail // All formats carry their tarball references directly on the doc. router.get('/:id', async (req, res, next) => { try { - const filter = artefactAccessFilter(await resolveCaller(req)) - const artefact = await getArtefact(req.params.id, filter) + const caller = await resolveCaller(req) + const artefact = await getArtefact(req.params.id, artefactAccessFilter(caller)) if (!artefact) throw httpError(404, 'artefact not found') - res.json(artefact) + res.json(stripScan(artefact, caller)) } catch (err) { next(err) } }) @@ -241,7 +272,8 @@ router.post('/file/:name', async (req, res, next) => { : { internal: true } }) stagingStored = false - res.status(201).json({ artefact }) + // Uploaders are never admin session callers; never leak scan data. + res.status(201).json({ artefact: omitScan(artefact) }) } catch (err) { if (stagingStored) await filesStorage.delete(stagingPath).catch(() => {}) next(err) @@ -299,7 +331,8 @@ router.post('/npm/:id', async (req, res, next) => { existing }) stagingStored = false - res.status(201).json({ artefact }) + // Uploaders are never admin session callers; never leak scan data. + res.status(201).json({ artefact: omitScan(artefact) }) } catch (err) { if (stagingStored) await filesStorage.delete(stagingPath).catch(() => {}) next(err) diff --git a/api/src/artefacts/service.ts b/api/src/artefacts/service.ts index 2699d4e..3014a35 100644 --- a/api/src/artefacts/service.ts +++ b/api/src/artefacts/service.ts @@ -10,6 +10,7 @@ import mongo from '#mongo' import config from '#config' import { filesStorage } from '../files-storage/index.ts' import { extractManifest, type Manifest } from './operations.ts' +import { enqueueScan } from '../scanning/service.ts' export type { Manifest, ExtractManifestResult } from './operations.ts' @@ -49,6 +50,63 @@ export const listGroupValues = async ( .sort((a, b) => a.localeCompare(b)) } +// --- scan summary (fleet-wide, admin dashboard) --------------------------- + +export type ScanSummaryResult = { + enabled: boolean + totals: { critical: number, high: number, medium: number, low: number, unknown: number, artefactsWithCritical: number } + health: { npmTotal: number, scanned: number, error: number, pending: number, never: number, oldestScanAt: string | null } + worstOffenders: { _id: string, name: string, status?: string, summary?: { critical?: number, high?: number, medium?: number, low?: number, unknown?: number, total?: number } }[] +} + +// Fleet-wide vulnerability roll-up over npm artefacts, in a single $facet. +// Artefacts without a scan contribute 0 to totals and to the "never" bucket. +export const getScanSummary = async (): Promise => { + const enabled = config.scanning?.enabled ?? false + const num = (path: string) => ({ $ifNull: [path, 0] }) + const [facet] = await mongo.artefacts.aggregate([ + { $match: { format: 'npm' } }, + { + $facet: { + totals: [{ + $group: { + _id: null, + critical: { $sum: num('$scan.summary.critical') }, + high: { $sum: num('$scan.summary.high') }, + medium: { $sum: num('$scan.summary.medium') }, + low: { $sum: num('$scan.summary.low') }, + unknown: { $sum: num('$scan.summary.unknown') }, + artefactsWithCritical: { $sum: { $cond: [{ $gt: [num('$scan.summary.critical'), 0] }, 1, 0] } } + } + }], + health: [{ + $group: { + _id: null, + npmTotal: { $sum: 1 }, + scanned: { $sum: { $cond: [{ $eq: ['$scan.status', 'success'] }, 1, 0] } }, + error: { $sum: { $cond: [{ $eq: ['$scan.status', 'error'] }, 1, 0] } }, + pending: { $sum: { $cond: [{ $in: ['$scan.status', ['pending', 'running']] }, 1, 0] } }, + never: { $sum: { $cond: [{ $eq: [{ $ifNull: ['$scan.status', null] }, null] }, 1, 0] } }, + oldestScanAt: { $min: { $cond: [{ $eq: ['$scan.status', 'success'] }, '$scan.finishedAt', null] } } + } + }], + worstOffenders: [ + { $match: { 'scan.summary.total': { $gt: 0 } } }, + { $sort: { 'scan.summary.critical': -1, 'scan.summary.high': -1 } }, + { $limit: 10 }, + { $project: { _id: 1, name: 1, status: '$scan.status', summary: '$scan.summary' } } + ] + } + } + ]).toArray() as any[] + + const totals = facet?.totals?.[0] ?? { critical: 0, high: 0, medium: 0, low: 0, unknown: 0, artefactsWithCritical: 0 } + delete totals._id + const health = facet?.health?.[0] ?? { npmTotal: 0, scanned: 0, error: 0, pending: 0, never: 0, oldestScanAt: null } + delete health._id + return { enabled, totals, health, worstOffenders: facet?.worstOffenders ?? [] } +} + // --- metadata patch ------------------------------------------------------- export const patchArtefact = (id: string, body: Record) => { @@ -72,6 +130,9 @@ export const deleteArtefact = async (artefact: Artefact) => { await mongo.artefacts.deleteOne({ _id: artefact._id }) if (artefact.path) await filesStorage.delete(artefact.path).catch(() => {}) await mongo.thumbnails.deleteMany({ artefactId: artefact._id }) + // Drop the full-findings doc so a later artefact reusing this id can't surface + // stale scan results before its own scan runs. + await mongo.artefactScans.deleteOne({ _id: artefact._id }) } // --- npm upload ----------------------------------------------------------- @@ -143,6 +204,9 @@ export const commitNpmUpload = async (params: { await filesStorage.delete(existing.path).catch(() => {}) } + // Advisory scan runs in the background; never blocks the upload response. + await enqueueScan(id) + return (await mongo.artefacts.findOne({ _id: id }))! } diff --git a/api/src/config.ts b/api/src/config.ts index d36308a..368cb06 100644 --- a/api/src/config.ts +++ b/api/src/config.ts @@ -1,7 +1,17 @@ import type { ApiConfig } from '../config/type/index.ts' import { assertValid } from '../config/type/index.ts' import config from 'config' +import { tmpdir } from 'node:os' +import { join } from 'node:path' assertValid(config, { lang: 'en', name: 'config', internal: true }) -export default config as ApiConfig +const typedConfig = config as ApiConfig + +// Resolved working temp dir (mirrors the processings convention): an explicit +// tmpDir, else /tmp, else an OS temp fallback. The scan cache lives +// under /scan-cache. Mount tmpDir as an emptyDir in k8s. +export const tmpDir = typedConfig.tmpDir ?? + (typedConfig.dataDir ? join(typedConfig.dataDir, 'tmp') : join(tmpdir(), 'data-fair-registry')) + +export default typedConfig diff --git a/api/src/mongo.ts b/api/src/mongo.ts index b659a79..211b53f 100644 --- a/api/src/mongo.ts +++ b/api/src/mongo.ts @@ -17,6 +17,32 @@ export type Thumbnail = { createdAt: string } +export type ScanSeverity = 'critical' | 'high' | 'medium' | 'low' | 'unknown' + +export type ScanFinding = { + id: string + pkgName: string + installedVersion: string + fixedVersion?: string + severity: ScanSeverity + title?: string + primaryUrl?: string +} + +export type ScanLicense = { + pkgName: string + license: string +} + +export type ArtefactScan = { + _id: string // artefact id + scannedAt: string + scannerVersion: string + vulnDbUpdatedAt?: string + vulnerabilities: ScanFinding[] + licenses?: ScanLicense[] +} + export class RegistryMongo { get client () { return mongoLib.client @@ -42,6 +68,10 @@ export class RegistryMongo { return mongoLib.db.collection('thumbnails') } + get artefactScans () { + return mongoLib.db.collection('artefact-scans') + } + get remoteRegistries () { return mongoLib.db.collection('remote-registries') } diff --git a/api/src/scanning/cache.ts b/api/src/scanning/cache.ts new file mode 100644 index 0000000..7192294 --- /dev/null +++ b/api/src/scanning/cache.ts @@ -0,0 +1,72 @@ +import { readFile, writeFile, mkdir, rm, rename, readdir } from 'node:fs/promises' +import { join } from 'node:path' +import type { Readable } from 'node:stream' +import { extractTarballToDir } from './extract.ts' + +export type CacheMeta = { path: string, dataUpdatedAt?: string } +export type ArtefactRef = { artefactId: string, path: string, dataUpdatedAt?: string } +export type OpenTarball = (path: string) => Promise + +const slotDir = (cacheDir: string, artefactId: string) => + join(cacheDir, encodeURIComponent(artefactId)) + +// Return the extracted directory for an artefact, (re)extracting only when the +// stored bytes differ from the cached slot. Change is detected via the +// artefact's storage `path`, which carries a fresh randomUUID on every upload. +export const ensureExtracted = async ( + ref: ArtefactRef, + cacheDir: string, + openTarball: OpenTarball, + maxEntries: number +): Promise => { + const extractDir = slotDir(cacheDir, ref.artefactId) + const metaPath = join(extractDir, '.meta.json') + + // Cache hit: the cached slot was built from the same stored bytes. + try { + const meta = JSON.parse(await readFile(metaPath, 'utf-8')) as CacheMeta + if (meta.path === ref.path) return extractDir + } catch { /* missing/corrupt meta → treat as a miss */ } + + // Miss/changed: extract into a per-pid temp dir, then atomically swap it in. + const tmp = `${extractDir}.tmp.${process.pid}` + await rm(tmp, { recursive: true, force: true }) + await mkdir(tmp, { recursive: true }) + try { + await extractTarballToDir(await openTarball(ref.path), tmp, { maxEntries }) + const meta: CacheMeta = { path: ref.path, dataUpdatedAt: ref.dataUpdatedAt } + await writeFile(join(tmp, '.meta.json'), JSON.stringify(meta)) + } catch (err) { + await rm(tmp, { recursive: true, force: true }).catch(() => {}) + throw err + } + // Drop the stale slot only once the new one is fully built. + await rm(extractDir, { recursive: true, force: true }) + await rename(tmp, extractDir) + return extractDir +} + +// Matches the `.tmp.` suffix that ensureExtracted appends to a slot name. +// Anchored to a trailing numeric pid so it can't be confused with a slot whose +// (encoded) id merely contains ".tmp." somewhere in the middle. +const TMP_DIR_SUFFIX = /\.tmp\.\d+$/ + +// Remove cache slots whose artefact id is no longer present. Skips in-flight +// `.tmp.` extraction dirs (deleting one could break a concurrent +// ensureExtracted; orphans are reclaimed when the emptyDir resets on restart). +export const pruneExtracted = async (cacheDir: string, validIds: Set): Promise => { + let entries: string[] + try { + entries = await readdir(cacheDir) + } catch { + return // cache dir doesn't exist yet → nothing to prune + } + for (const entry of entries) { + if (TMP_DIR_SUFFIX.test(entry)) continue + let id: string + try { id = decodeURIComponent(entry) } catch { continue } + if (!validIds.has(id)) { + await rm(join(cacheDir, entry), { recursive: true, force: true }).catch(() => {}) + } + } +} diff --git a/api/src/scanning/extract.ts b/api/src/scanning/extract.ts new file mode 100644 index 0000000..f307ca4 --- /dev/null +++ b/api/src/scanning/extract.ts @@ -0,0 +1,35 @@ +import type { Readable } from 'node:stream' +import { pipeline } from 'node:stream/promises' +import * as tar from 'tar' +import { httpError } from '@data-fair/lib-utils/http-errors.js' + +export type ExtractOpts = { maxEntries: number } + +// Extract a gzipped npm tarball stream into `dir`. node-tar handles gunzip and +// guards against path traversal (it strips `..` and refuses absolute paths by +// default). An entry counter caps the file count to mirror upload-time limits. +// +// Implementation note: node-tar swallows throws from the `filter` callback, so +// instead of throwing we count in `filter` and return false once the cap is +// exceeded — that stops further entries from being written to disk — then +// reject after the pipeline drains. This bounds on-disk output to the cap +// rather than extracting everything and only complaining afterwards. +export const extractTarballToDir = async (stream: Readable, dir: string, opts: ExtractOpts): Promise => { + let count = 0 + let capError: Error | null = null + const extract = tar.x({ + cwd: dir, + gzip: true, + filter: (path: string) => { + count++ + if (count > opts.maxEntries) { + capError ??= httpError(413, `tarball exceeds ${opts.maxEntries} entries`) + return false // skip this and every subsequent entry + } + // Reject anything outside cwd; node-tar already strips '..' but be explicit. + return !path.includes('..') + } + }) + await pipeline(stream, extract) + if (capError) throw capError +} diff --git a/api/src/scanning/operations.ts b/api/src/scanning/operations.ts new file mode 100644 index 0000000..990950a --- /dev/null +++ b/api/src/scanning/operations.ts @@ -0,0 +1,114 @@ +import type { ScanFinding, ScanSeverity, ScanLicense } from '#mongo' + +// --- severity --------------------------------------------------------------- + +const LABEL_TO_BUCKET: Record = { + CRITICAL: 'critical', + HIGH: 'high', + MODERATE: 'medium', + MEDIUM: 'medium', + LOW: 'low' +} + +// osv-scanner exposes severity inconsistently: a textual label in +// vulnerabilities[].database_specific.severity, and/or a numeric CVSS score +// in groups[].max_severity. Prefer the label, fall back to CVSS banding. +export const severityBucket = (label?: string, cvss?: string): ScanSeverity => { + if (label && LABEL_TO_BUCKET[label.toUpperCase()]) return LABEL_TO_BUCKET[label.toUpperCase()] + const score = cvss !== undefined ? Number(cvss) : NaN + if (!Number.isNaN(score)) { + if (score >= 9) return 'critical' + if (score >= 7) return 'high' + if (score >= 4) return 'medium' + if (score > 0) return 'low' + } + return 'unknown' +} + +export type Summary = { critical: number, high: number, medium: number, low: number, unknown: number, total: number } + +export const summarize = (findings: ScanFinding[]): Summary => { + const s: Summary = { critical: 0, high: 0, medium: 0, low: 0, unknown: 0, total: 0 } + for (const f of findings) { s[f.severity]++; s.total++ } + return s +} + +// --- osv json shape (subset we read) --------------------------------------- + +type OsvVuln = { + id: string + summary?: string + database_specific?: { severity?: string } + references?: { type?: string, url?: string }[] + affected?: { package?: { ecosystem?: string }, ranges?: { events?: { fixed?: string }[] }[] }[] +} +type OsvGroup = { ids?: string[], max_severity?: string } +type OsvPackage = { + package: { name: string, version: string, ecosystem?: string } + vulnerabilities?: OsvVuln[] + groups?: OsvGroup[] + licenses?: string[] +} +type OsvOutput = { results?: { packages?: OsvPackage[] }[] } + +const firstFixedVersion = (vuln: OsvVuln): string | undefined => { + for (const aff of vuln.affected ?? []) { + // Guard against cross-ecosystem ranges leaking a misleading "fixed" version. + const ecosystem = aff.package?.ecosystem + if (ecosystem && ecosystem !== 'npm') continue + for (const range of aff.ranges ?? []) { + for (const ev of range.events ?? []) { + if (ev.fixed) return ev.fixed + } + } + } + return undefined +} + +const primaryUrl = (vuln: OsvVuln): string | undefined => { + const ref = (vuln.references ?? []).find(r => r.type === 'ADVISORY') ?? (vuln.references ?? [])[0] + return ref?.url +} + +// Whether a (top-level) package.json declares an install lifecycle script. +// Install/preinstall/postinstall run automatically on `npm install` and are a +// primary supply-chain execution vector — surfaced as an advisory flag. +export const detectInstallScripts = (pkgJson: unknown): boolean => { + if (!pkgJson || typeof pkgJson !== 'object') return false + const scripts = (pkgJson as { scripts?: unknown }).scripts + if (!scripts || typeof scripts !== 'object') return false + for (const hook of ['install', 'preinstall', 'postinstall'] as const) { + if (typeof (scripts as Record)[hook] === 'string') return true + } + return false +} + +export const mapOsvOutput = (raw: unknown): { vulnerabilities: ScanFinding[], licenses: ScanLicense[], summary: Summary } => { + const out = (raw ?? {}) as OsvOutput + const findings: ScanFinding[] = [] + const licenses: ScanLicense[] = [] + for (const result of out.results ?? []) { + for (const pkg of result.packages ?? []) { + // CVSS score for the package's worst group, used as fallback severity. + const maxCvss = (pkg.groups ?? []) + .map(g => g.max_severity) + .filter((v): v is string => typeof v === 'string') + .sort((a, b) => Number(b) - Number(a))[0] + for (const vuln of pkg.vulnerabilities ?? []) { + findings.push({ + id: vuln.id, + pkgName: pkg.package.name, + installedVersion: pkg.package.version, + fixedVersion: firstFixedVersion(vuln), + severity: severityBucket(vuln.database_specific?.severity, maxCvss), + title: vuln.summary, + primaryUrl: primaryUrl(vuln) + }) + } + for (const lic of pkg.licenses ?? []) { + if (lic && lic !== 'UNKNOWN') licenses.push({ pkgName: pkg.package.name, license: lic }) + } + } + } + return { vulnerabilities: findings, licenses, summary: summarize(findings) } +} diff --git a/api/src/scanning/router.ts b/api/src/scanning/router.ts new file mode 100644 index 0000000..023d1b5 --- /dev/null +++ b/api/src/scanning/router.ts @@ -0,0 +1,36 @@ +import { Router } from 'express' +import { session } from '@data-fair/lib-express/index.js' +import { httpError } from '@data-fair/lib-utils/http-errors.js' +import mongo from '#mongo' +import config from '#config' +import { getArtefactById } from '../artefacts/service.ts' +import { enqueueScan } from './service.ts' + +// Mounted at /api/v1/artefacts/:id/scan (mergeParams to read :id). +const router = Router({ mergeParams: true }) +export default router + +// Full findings (admin only). +router.get('/', async (req, res, next) => { + try { + await session.reqAdminMode(req) + const id = (req.params as { id: string }).id + const scan = await mongo.artefactScans.findOne({ _id: id }) + if (!scan) throw httpError(404, 'no scan for this artefact') + res.json(scan) + } catch (err) { next(err) } +}) + +// Trigger an on-demand (re)scan (admin only). +router.post('/', async (req, res, next) => { + try { + await session.reqAdminMode(req) + if (!config.scanning?.enabled) throw httpError(503, 'scanning is not enabled on this deployment') + const id = (req.params as { id: string }).id + const artefact = await getArtefactById(id) + if (!artefact) throw httpError(404, 'artefact not found') + if (artefact.format !== 'npm') throw httpError(400, 'only npm artefacts can be scanned') + await enqueueScan(id) + res.status(202).json({ status: 'pending' }) + } catch (err) { next(err) } +}) diff --git a/api/src/scanning/runner.ts b/api/src/scanning/runner.ts new file mode 100644 index 0000000..4c83204 --- /dev/null +++ b/api/src/scanning/runner.ts @@ -0,0 +1,148 @@ +import { spawn } from 'node:child_process' +import { readFile, writeFile, mkdtemp, rm, mkdir, stat } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import type { ChildProcess } from 'node:child_process' +import config from '#config' +import type { ScanFinding, ScanLicense } from '#mongo' +import { mapOsvOutput, summarize, detectInstallScripts, type Summary } from './operations.ts' + +export type ScanResult = { + vulnerabilities: ScanFinding[] + licenses: ScanLicense[] + summary: Summary + hasInstallScripts: boolean + scannerVersion: string + // mtime of the local OSV DB the scan ran against, so admins can gauge + // result freshness. Undefined if the DB file can't be stat'd. + vulnDbUpdatedAt?: string +} + +export interface Scanner { + // Scan an already-extracted artefact directory. + scanDir (dir: string): Promise + // Refresh the local offline DB (online). Folded into the periodic job. + refreshDb (): Promise + version (): Promise +} + +const run = (args: string[], timeoutMs: number): Promise<{ code: number, stdout: string, stderr: string }> => + new Promise((resolve, reject) => { + const scannerPath = config.scanning?.osvScannerPath ?? 'osv-scanner' + const child: ChildProcess = spawn(scannerPath, args, { timeout: timeoutMs }) + let stdout = '' + let stderr = '' + child.stdout!.on('data', (d: Buffer) => { stdout += d }) + child.stderr!.on('data', (d: Buffer) => { stderr += d }) + child.on('error', reject) + child.on('close', (code: number | null, signal: string | null) => { + // A timeout kills the child with a signal; treat that as a failure rather + // than silently resolving with empty stdout (which would look "clean"). + if (signal) { reject(new Error(`osv-scanner terminated by signal ${signal} (timeout?)`)); return } + resolve({ code: code ?? -1, stdout, stderr }) + }) + }) + +// osv-scanner exits 0 = no vulns, 1 = vulns found, >1 = real error +// (e.g. 128 = "no package sources found"). Flags pinned by the Task 1 spike +// against osv-scanner v2.2.3. +const EXIT_VULNS_FOUND = 1 + +// The scalibr package.json extractor is NOT enabled by default; without it a +// directory scan of bundled (lockfile-less) node_modules finds nothing. +const PLUGIN_ARGS = ['--experimental-plugins', 'javascript/packagejson'] + +class OsvScanner implements Scanner { + // The binary doesn't change at runtime, so resolve the version once. + private versionCache?: string + + async version (): Promise { + if (this.versionCache) return this.versionCache + const { stdout } = await run(['--version'], 30_000) + // First line: "osv-scanner version: X.Y.Z" + this.versionCache = stdout.trim().split('\n')[0] || 'unknown' + return this.versionCache + } + + // Best-effort freshness timestamp of the local OSV DB (npm/all.zip mtime). + private async dbUpdatedAt (): Promise { + const dbDir = config.scanning?.dbDir ?? 'osv-db' + try { + const { mtime } = await stat(join(dbDir, 'osv-scanner', 'npm', 'all.zip')) + return mtime.toISOString() + } catch { return undefined } + } + + async refreshDb (): Promise { + // The DB is only downloaded when >=1 package is found, so scan a tiny dummy + // dir holding one package.json (with --allow-no-lockfiles). + const dbDir = config.scanning?.dbDir ?? 'osv-db' + const timeoutMs = (config.scanning?.timeoutSeconds ?? 300) * 1000 + await mkdir(dbDir, { recursive: true }) + const dummy = await mkdtemp(join(tmpdir(), 'osv-db-refresh-')) + try { + await mkdir(join(dummy, 'node_modules', 'left-pad'), { recursive: true }) + await writeFile(join(dummy, 'node_modules', 'left-pad', 'package.json'), '{"name":"left-pad","version":"1.0.0"}') + const args = [ + 'scan', 'source', '--recursive', + // Scan extracted artefacts regardless of any .gitignore: the cache dir + // may live under a git-ignored path (e.g. dev's ./data), and a tarball + // could even bundle its own .gitignore. Both would otherwise hide files. + '--no-ignore', + ...PLUGIN_ARGS, + '--offline-vulnerabilities', + '--download-offline-databases', + '--local-db-path', dbDir, + '--allow-no-lockfiles', + '--format', 'json', + dummy + ] + const { code, stderr } = await run(args, timeoutMs) + if (code > EXIT_VULNS_FOUND) throw new Error(`osv-scanner db refresh failed (exit ${code}): ${stderr.slice(0, 500)}`) + } finally { + await rm(dummy, { recursive: true, force: true }).catch(() => {}) + } + } + + async scanDir (dir: string): Promise { + const dbDir = config.scanning?.dbDir ?? 'osv-db' + const timeoutMs = (config.scanning?.timeoutSeconds ?? 300) * 1000 + const args = [ + 'scan', 'source', '--recursive', + // Scan extracted artefacts regardless of any .gitignore: the cache dir + // lives under a git-ignored path (dev's ./data) and a tarball could even + // bundle its own .gitignore — both would otherwise hide all files. + '--no-ignore', + ...PLUGIN_ARGS, + '--offline-vulnerabilities', + '--local-db-path', dbDir, + '--allow-no-lockfiles', + '--format', 'json', + dir + ] + const { code, stdout, stderr } = await run(args, timeoutMs) + if (code > EXIT_VULNS_FOUND) { + throw new Error(`osv-scanner failed (exit ${code}): ${stderr.slice(0, 500)}`) + } + const raw = stdout.trim() ? JSON.parse(stdout) : { results: [] } + const { vulnerabilities, licenses, summary } = mapOsvOutput(raw) + + let hasInstallScripts = false + try { + const topPkg = JSON.parse(await readFile(join(dir, 'package', 'package.json'), 'utf-8')) + hasInstallScripts = detectInstallScripts(topPkg) + } catch { /* top-level package.json missing/unreadable — leave false */ } + + return { + vulnerabilities, + licenses, + summary, + hasInstallScripts, + scannerVersion: await this.version(), + vulnDbUpdatedAt: await this.dbUpdatedAt() + } + } +} + +export { summarize } +export const osvScanner: Scanner = new OsvScanner() diff --git a/api/src/scanning/service.ts b/api/src/scanning/service.ts new file mode 100644 index 0000000..d88a460 --- /dev/null +++ b/api/src/scanning/service.ts @@ -0,0 +1,154 @@ +import { join } from 'node:path' +import locks from '@data-fair/lib-node/locks.js' +import { internalError } from '@data-fair/lib-node/observer.js' +import mongo from '#mongo' +import config, { tmpDir } from '#config' +import { filesStorage } from '../files-storage/index.ts' +import { ensureExtracted, pruneExtracted } from './cache.ts' +import { osvScanner, type Scanner } from './runner.ts' +import type { Summary } from './operations.ts' + +let scanner: Scanner = osvScanner +// Test seam: swap the scanner implementation (no production caller). +export const __setScanner = (s: Scanner) => { scanner = s } + +// Scanning is a security control, so its activity is logged to stdout (visible +// in container logs) at every step — queue, start, per-artefact result, db +// refresh, and the periodic rescan bounds — not just on failure. Errors also go +// through internalError() so they increment the Prometheus counter. +const log = (msg: string) => console.log(`[scan] ${msg}`) +const durationSec = (since: number) => ((Date.now() - since) / 1000).toFixed(1) + +// One-line, grep-able summary of a scan result for the logs. +const formatSummary = (s: Summary, hasInstallScripts: boolean): string => { + const parts: string[] = [] + if (s.total === 0) { + parts.push('no vulnerabilities') + } else { + const sev = (['critical', 'high', 'medium', 'low', 'unknown'] as const) + .filter(k => s[k] > 0).map(k => `${k}=${s[k]}`).join(' ') + parts.push(`${s.total} vulnerabilit${s.total === 1 ? 'y' : 'ies'} (${sev})`) + } + if (hasInstallScripts) parts.push('install-scripts=yes') + return parts.join(', ') +} + +// The extracted-mirror cache lives under the configured temp dir. +const scanCacheDir = join(tmpDir, 'scan-cache') +const openTarball = (path: string) => filesStorage.readStream(path).then(r => r.body) + +// In-process concurrency gate. +let active = 0 +const waiters: (() => void)[] = [] +const acquireSlot = async () => { + if (active >= (config.scanning?.concurrency ?? 1)) { + await new Promise(resolve => waiters.push(resolve)) + } + active++ +} +const releaseSlot = () => { + active-- + const next = waiters.shift() + if (next) next() +} + +const setStatus = (id: string, scan: Record) => + mongo.artefacts.updateOne({ _id: id }, { $set: { scan } }) + +// Mark an artefact pending and kick a background scan. Never throws to the +// caller; failures are recorded on the doc. +export const enqueueScan = async (id: string): Promise => { + if (!config.scanning?.enabled) return + // Dotted paths so a re-queue keeps the previously-known summary visible until + // the new scan finishes, rather than blanking it while pending. + await mongo.artefacts.updateOne( + { _id: id }, + { $set: { 'scan.status': 'pending', 'scan.queuedAt': new Date().toISOString() } } + ) + log(`${id}: scan queued`) + // Fire-and-forget; do not block the upload response. + runScanNow(id).catch(err => internalError('scan', err)) +} + +// Run a full scan synchronously (used by the background task + rescanAll). +export const runScanNow = async (id: string, opts: { refreshDb?: boolean } = {}): Promise => { + if (!config.scanning?.enabled) return + const lockId = `scan-${id}` + if (!await locks.acquire(lockId)) return // another instance/worker has it + await acquireSlot() + const startedAt = Date.now() + try { + const artefact = await mongo.artefacts.findOne({ _id: id }) + if (!artefact || artefact.format !== 'npm' || !artefact.path) return + + log(`${id}: scan started${opts.refreshDb ? ' (with db refresh)' : ''}`) + await setStatus(id, { ...artefact.scan, status: 'running', startedAt: new Date().toISOString() }) + if (opts.refreshDb) await scanner.refreshDb() + + // Reuse the cached extraction when the stored bytes are unchanged. + const dir = await ensureExtracted( + { artefactId: id, path: artefact.path, dataUpdatedAt: artefact.dataUpdatedAt }, + scanCacheDir, + openTarball, + config.maxTarEntries ?? 100000 + ) + + const result = await scanner.scanDir(dir) + const now = new Date().toISOString() + + await mongo.artefactScans.replaceOne( + { _id: id }, + { + scannedAt: now, + scannerVersion: result.scannerVersion, + ...(result.vulnDbUpdatedAt ? { vulnDbUpdatedAt: result.vulnDbUpdatedAt } : {}), + vulnerabilities: result.vulnerabilities, + ...(result.licenses.length ? { licenses: result.licenses } : {}) + }, + { upsert: true } + ) + await setStatus(id, { + status: 'success', + finishedAt: now, + scannerVersion: result.scannerVersion, + ...(result.vulnDbUpdatedAt ? { vulnDbUpdatedAt: result.vulnDbUpdatedAt } : {}), + hasInstallScripts: result.hasInstallScripts, + summary: result.summary + }) + log(`${id}: ${formatSummary(result.summary, result.hasInstallScripts)} in ${durationSec(startedAt)}s`) + } catch (err) { + // A scan failure is itself security-relevant (the artefact's safety is now + // unknown): log it with the artefact id AND bump the error counter. + internalError('scan', err, `(artefact ${id})`) + await setStatus(id, { status: 'error', finishedAt: new Date().toISOString(), error: (err as Error).message?.slice(0, 500) }) + .catch(e => internalError('scan', e)) + } finally { + releaseSlot() + await locks.release(lockId) + } +} + +// Periodic job: refresh the DB once, then rescan every npm artefact. +export const rescanAll = async (): Promise => { + if (!config.scanning?.enabled) return + const startedAt = Date.now() + log('rescan started: refreshing OSV database') + try { + const dbStart = Date.now() + await scanner.refreshDb() + log(`OSV database refreshed in ${durationSec(dbStart)}s`) + } catch (err) { + internalError('scan-db-refresh', err) + } + const ids = await mongo.artefacts.find({ format: 'npm' }, { projection: { _id: 1 } }).toArray() + log(`rescan: scanning ${ids.length} npm artefact(s)`) + // Drop cached extractions for artefacts that no longer exist (runs on every + // pod, so each self-prunes its own emptyDir). The id snapshot is taken just + // above: an artefact uploaded mid-rescan may have its fresh slot pruned once + // and simply re-extract on its next scan — harmless and self-healing. + await pruneExtracted(scanCacheDir, new Set(ids.map(a => a._id))).catch(err => internalError('scan-prune', err)) + for (const { _id } of ids) { + await runScanNow(_id).catch(err => internalError('scan', err)) + } + log(`rescan finished: ${ids.length} artefact(s) in ${durationSec(startedAt)}s`) +} diff --git a/api/src/server.ts b/api/src/server.ts index 3b5b967..ba504da 100644 --- a/api/src/server.ts +++ b/api/src/server.ts @@ -9,6 +9,7 @@ import { app } from './app.ts' import config from '#config' import mongo from '#mongo' import { syncAllRemoteRegistries } from './remote-registries/sync.ts' +import { rescanAll } from './scanning/service.ts' import { renameFilePathToPath } from './boot-rename-file-path.ts' const server = createServer(app) @@ -22,6 +23,7 @@ server.requestTimeout = 60 * 60 * 1000 server.on('timeout', () => internalError('http-timeout', 'http socket timeout')) server.on('clientError', (err) => internalError('http-client-error', err)) let syncTimer: ReturnType | undefined +let rescanTimer: ReturnType | undefined export const start = async () => { if (config.observer?.active) await startObserver(config.observer.port) @@ -48,11 +50,20 @@ export const start = async () => { }) }, 24 * 60 * 60 * 1000) + if (config.scanning?.enabled) { + // Warm the DB + do an initial pass shortly after boot, then on the interval. + rescanAll().catch(err => internalError('rescan-all', err)) + rescanTimer = setInterval(() => { + rescanAll().catch(err => internalError('rescan-all', err)) + }, (config.scanning.rescanIntervalHours ?? 24) * 60 * 60 * 1000) + } + console.log(`API server listening on port ${config.port}`) } export const stop = async () => { if (syncTimer) clearInterval(syncTimer) + if (rescanTimer) clearInterval(rescanTimer) await httpTerminator.terminate() if (config.observer?.active) await stopObserver() await locks.stop() diff --git a/api/src/ui-config.ts b/api/src/ui-config.ts index ba61492..f5490b6 100644 --- a/api/src/ui-config.ts +++ b/api/src/ui-config.ts @@ -1,4 +1,10 @@ -export const uiConfig = {} +import config from '#config' + +// Injected into the SPA as window.__UI_CONFIG and read via $uiConfig in +// ui/src/context.ts. Keep this minimal and non-secret. +export const uiConfig = { + scanning: { enabled: config.scanning?.enabled ?? false } +} export type UiConfig = typeof uiConfig export default uiConfig diff --git a/api/types/artefact/schema.js b/api/types/artefact/schema.js index abbb343..51b7a41 100644 --- a/api/types/artefact/schema.js +++ b/api/types/artefact/schema.js @@ -154,6 +154,36 @@ export default { // (lib-node) use it to decide whether to run `npm rebuild` after // extraction. hasNativeModules: { type: 'boolean', readOnly: true }, + // Advisory vulnerability-scan summary. Admin-only: stripped from + // responses for non-admin callers in the artefacts router. Full + // findings live in the separate `artefactScans` collection. + scan: { + type: 'object', + readOnly: true, + additionalProperties: false, + properties: { + status: { type: 'string', enum: ['pending', 'running', 'success', 'error'] }, + queuedAt: { type: 'string', format: 'date-time' }, + startedAt: { type: 'string', format: 'date-time' }, + finishedAt: { type: 'string', format: 'date-time' }, + scannerVersion: { type: 'string' }, + vulnDbUpdatedAt: { type: 'string', format: 'date-time' }, + hasInstallScripts: { type: 'boolean' }, + error: { type: 'string' }, + summary: { + type: 'object', + additionalProperties: false, + properties: { + critical: { type: 'integer' }, + high: { type: 'integer' }, + medium: { type: 'integer' }, + low: { type: 'integer' }, + unknown: { type: 'integer' }, + total: { type: 'integer' } + } + } + } + }, uploadedBy: { type: 'object', readOnly: true, diff --git a/docs/superpowers/plans/2026-06-04-npm-vulnerability-scanning.md b/docs/superpowers/plans/2026-06-04-npm-vulnerability-scanning.md new file mode 100644 index 0000000..2cfe7cb --- /dev/null +++ b/docs/superpowers/plans/2026-06-04-npm-vulnerability-scanning.md @@ -0,0 +1,1512 @@ +# npm Vulnerability Scanning Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add advisory, admin-only vulnerability scanning of `npm`-format artefacts (bundled-`node_modules` plugin tarballs) using osv-scanner, surfacing dependency CVEs (plus licenses and an install-script flag) to administrators without ever blocking uploads or downloads. + +**Architecture:** A self-contained `api/src/scanning/` module shells out to a bundled **osv-scanner v2** binary against each tarball extracted to an ephemeral temp dir, in **offline local-DB mode**. Scans run async on upload, on a daily interval, and on-demand; concurrency is gated in-process and deduped across instances with the existing MongoDB distributed lock. A summary rides on the artefact doc (admin-only); full findings live in a new `artefactScans` collection. + +**Tech Stack:** Node 24, Express 5, MongoDB (`@data-fair/lib-node`), TypeScript, JSON-schema-generated types (`npm run build-types`), Playwright test runner (unit/api/e2e projects), Vue 3 + Vuetify (UI), osv-scanner v2 + node-tar. + +**Reference spec:** `docs/superpowers/specs/2026-06-04-npm-vulnerability-scanning-design.md` + +--- + +## File Structure + +**Created:** +- `api/src/scanning/operations.ts` — pure helpers: osv JSON → internal shape mapper, severity bucketing, install-script detection. Unit-testable, no I/O. +- `api/src/scanning/extract.ts` — extract a gzipped tarball stream to a temp dir (node-tar, entry-capped). +- `api/src/scanning/runner.ts` — `Scanner` interface + the osv-scanner subprocess implementation (build argv, spawn, parse via `operations.ts`). +- `api/src/scanning/service.ts` — stateful orchestration: enqueue, in-process semaphore, distributed lock, persistence, `enqueueScan` / `runScanNow` / `rescanAll` / `refreshDb`. +- `api/src/scanning/router.ts` — `GET /:id/scan`, `POST /:id/scan` (mounted as a sub-router on the artefacts router, like thumbnails). +- `tests/scanning-operations.unit.spec.ts` — unit tests for `operations.ts`. +- `tests/scanning-extract.unit.spec.ts` — unit test for `extract.ts`. +- `tests/scanning.api.spec.ts` — API tests for endpoints + admin-only stripping (uses a test-env injection helper). +- `tests/resources/osv-sample-output.json` — captured real osv-scanner v2 JSON, the fixture for the mapper test (produced in Task 1). + +**Modified:** +- `api/config/type/schema.json` — add `scanning` config block. +- `api/config/default.js` — defaults for `scanning`. +- `api/config/custom-environment-variables.js` — env var bindings for `scanning`. +- `api/types/artefact/schema.js` — add readOnly `scan` summary field. +- `api/src/mongo.ts` — `artefactScans` collection getter + `ArtefactScan` type + index. +- `api/src/artefacts/service.ts` — enqueue a scan at the end of `commitNpmUpload`. +- `api/src/artefacts/router.ts` — mount the scanning sub-router; strip `scan` for non-admin callers on list + detail. +- `api/src/server.ts` — periodic rescan interval + DB warm-up on boot. +- `api/src/app.ts` — test-env helper route to inject scan state. +- `ui/src/components/artefact-admin.vue` — admin scan panel (status, findings table, rescan button). +- `ui/src/i18n/*` (wherever artefact-admin strings live) — labels. +- `Dockerfile` — install the osv-scanner v2 binary. +- `AGENTS.md` — short note on the scanning feature + osv-scanner dependency. + +--- + +## Task 1: Spike — confirm osv-scanner detects bundled node_modules (make-or-break) + +This de-risks the entire feature and pins the exact CLI invocation + JSON shape that Tasks 4 and 8 depend on. Do this first; do not proceed if it fails without adopting the SBOM fallback. + +**Files:** +- Create: `tests/resources/osv-sample-output.json` + +- [ ] **Step 1: Install osv-scanner v2 locally for the spike** + +Run (Linux x86_64): +```bash +OSV_VERSION=v2.2.3 +curl -sSfL -o /tmp/osv-scanner "https://github.com/google/osv-scanner/releases/download/${OSV_VERSION}/osv-scanner_linux_amd64" +chmod +x /tmp/osv-scanner +/tmp/osv-scanner --version +``` +Expected: prints a version `>= 2.x`. (If the asset name differs for the pinned release, list assets with `gh release view ${OSV_VERSION} --repo google/osv-scanner`.) + +- [ ] **Step 2: Build a realistic bundled-deps fixture tarball with a KNOWN-vulnerable dependency, no lockfile** + +Run: +```bash +WORK=$(mktemp -d) +mkdir -p "$WORK/package/node_modules/minimist" +# Top-level plugin manifest (no package-lock.json, mirroring a published npm tarball) +cat > "$WORK/package/package.json" <<'JSON' +{ "name": "@test/vuln-plugin", "version": "1.0.0", "scripts": { "postinstall": "node ./setup.js" } } +JSON +# A bundled dependency pinned to a version with a well-known advisory (minimist <0.2.1 / <1.2.3 prototype pollution) +cat > "$WORK/package/node_modules/minimist/package.json" <<'JSON' +{ "name": "minimist", "version": "0.0.8" } +JSON +tar -C "$WORK" -czf "$WORK/plugin.tgz" package +# Extract as the scanner will see it +mkdir -p "$WORK/extracted" && tar -C "$WORK/extracted" -xzf "$WORK/plugin.tgz" +echo "$WORK" +``` + +- [ ] **Step 3: Run osv-scanner against the EXTRACTED directory and confirm it finds the bundled dep** + +Run: +```bash +/tmp/osv-scanner scan source --recursive --experimental-plugins javascript/packagejson --format json "$WORK/extracted" > /tmp/osv-out.json; echo "exit=$?" +cat /tmp/osv-out.json +``` +Expected: exit code `1` (osv-scanner exits non-zero **when vulnerabilities are found** — this is normal, not an error), and the JSON `.results[].packages[]` contains an entry with `package.name == "minimist"`, `package.version == "0.0.8"`, and a non-empty `vulnerabilities` array. **This is the make-or-break check.** + +If `minimist` is absent from the output, the lockfile-less `node_modules` walk is not working in this version — STOP and switch to the SBOM fallback (generate an SBOM with `cdxgen -o bom.json ` then `osv-scanner scan --sbom bom.json`), updating Tasks 7/8 accordingly. Record the decision in the spec. + +- [ ] **Step 4: Confirm offline local-DB mode works and pin the exact flags** + +Run: +```bash +mkdir -p /tmp/osv-db +# Refresh DB (online, once): +/tmp/osv-scanner scan source --recursive --experimental-plugins javascript/packagejson --offline-vulnerabilities --download-offline-databases --local-db-path /tmp/osv-db --allow-no-lockfiles --format json "$WORK/extracted" > /tmp/osv-out2.json; echo "refresh exit=$?" +# Subsequent OFFLINE scan (no network) against the cached DB: +/tmp/osv-scanner scan source --recursive --experimental-plugins javascript/packagejson --offline-vulnerabilities --local-db-path /tmp/osv-db --allow-no-lockfiles --format json "$WORK/extracted" > /tmp/osv-out3.json; echo "offline exit=$?" +ls -la /tmp/osv-db +``` +Expected: the refresh populates `/tmp/osv-db`, and the offline scan still reports the `minimist` vulnerability. Record the exact working flag set in a comment in `runner.ts` later. (Flag names can drift between osv-scanner minor versions; whatever set works here is the source of truth for Task 8.) + +- [ ] **Step 5: Save the captured output as the mapper test fixture** + +Run: +```bash +cp /tmp/osv-out3.json tests/resources/osv-sample-output.json +# Sanity check it parses and has the expected package +node -e "const o=require('./tests/resources/osv-sample-output.json'); const pkgs=o.results.flatMap(r=>r.packages||[]); const m=pkgs.find(p=>p.package.name==='minimist'); if(!m||!(m.vulnerabilities||[]).length) throw new Error('fixture missing minimist vuln'); console.log('ok, severity sample:', JSON.stringify(m.groups||m.vulnerabilities[0].database_specific||{}))" +``` +Expected: prints `ok, severity sample: ...`. Inspect the printed shape — note where severity lives (`groups[].max_severity` CVSS string and/or `vulnerabilities[].database_specific.severity` label). Tasks 4's mapper is written against THIS file. + +- [ ] **Step 6: Commit the fixture and findings** + +```bash +git add tests/resources/osv-sample-output.json +git commit -m "test: capture osv-scanner v2 sample output for bundled node_modules (spike)" +``` + +--- + +## Task 2: Add the `scanning` config block + +**Files:** +- Modify: `api/config/type/schema.json` +- Modify: `api/config/default.js` +- Modify: `api/config/custom-environment-variables.js` + +- [ ] **Step 1: Add the `scanning` object to the config JSON schema** + +In `api/config/type/schema.json`, add this property inside `properties` (e.g. after `maxTarEntries`): +```json + "scanning": { + "type": "object", + "additionalProperties": false, + "default": {}, + "properties": { + "enabled": { + "type": "boolean", + "default": false, + "description": "Enable advisory vulnerability scanning of npm artefacts via osv-scanner. Off by default so deployments without the binary keep working." + }, + "osvScannerPath": { + "type": "string", + "default": "osv-scanner", + "description": "Path or command name of the osv-scanner v2 binary." + }, + "dbDir": { + "type": "string", + "default": "/data/osv-db", + "description": "Local directory holding the offline OSV vulnerability database." + }, + "concurrency": { + "type": "integer", + "minimum": 1, + "default": 1, + "description": "Maximum number of artefact scans running at once in this process." + }, + "rescanIntervalHours": { + "type": "integer", + "minimum": 1, + "default": 24, + "description": "Interval in hours between automatic rescans of all npm artefacts (also refreshes the OSV DB)." + }, + "timeoutSeconds": { + "type": "integer", + "minimum": 30, + "default": 300, + "description": "Per-artefact scan timeout in seconds (extraction + osv-scanner)." + } + } + }, +``` + +- [ ] **Step 2: Add defaults** + +In `api/config/default.js`, add after `maxTarEntries`: +```js + scanning: { + enabled: false, + osvScannerPath: 'osv-scanner', + dbDir: '/data/osv-db', + concurrency: 1, + rescanIntervalHours: 24, + timeoutSeconds: 300 + }, +``` + +- [ ] **Step 3: Add env var bindings** + +In `api/config/custom-environment-variables.js`, add after the `maxTarEntries` block: +```js + scanning: { + enabled: { + __name: 'SCANNING_ENABLED', + __format: 'boolean' + }, + osvScannerPath: 'OSV_SCANNER_PATH', + dbDir: 'OSV_DB_DIR', + concurrency: { + __name: 'SCANNING_CONCURRENCY', + __format: 'number' + }, + rescanIntervalHours: { + __name: 'SCANNING_RESCAN_INTERVAL_HOURS', + __format: 'number' + }, + timeoutSeconds: { + __name: 'SCANNING_TIMEOUT_SECONDS', + __format: 'number' + } + }, +``` + +- [ ] **Step 4: Regenerate config types** + +Run: `npm run build-types` +Expected: completes without error; `api/config/type/.type/index.d.ts` now includes a `scanning` property. + +- [ ] **Step 5: Verify types compile** + +Run: `npm run check-types` +Expected: PASS (no errors). + +- [ ] **Step 6: Commit** + +```bash +git add api/config +git commit -m "feat(config): add scanning config block (off by default)" +``` + +--- + +## Task 3: Add the `scan` summary field, `ArtefactScan` type, and Mongo collection + +**Files:** +- Modify: `api/types/artefact/schema.js` +- Modify: `api/src/mongo.ts` + +- [ ] **Step 1: Add the readOnly `scan` summary to the artefact schema** + +In `api/types/artefact/schema.js`, add inside `properties` (e.g. after `hasNativeModules`): +```js + // Advisory vulnerability-scan summary. Admin-only: stripped from + // responses for non-admin callers in the artefacts router. Full + // findings live in the separate `artefactScans` collection. + scan: { + type: 'object', + readOnly: true, + additionalProperties: false, + properties: { + status: { type: 'string', enum: ['pending', 'running', 'success', 'error'] }, + queuedAt: { type: 'string', format: 'date-time' }, + startedAt: { type: 'string', format: 'date-time' }, + finishedAt: { type: 'string', format: 'date-time' }, + scannerVersion: { type: 'string' }, + vulnDbUpdatedAt: { type: 'string', format: 'date-time' }, + hasInstallScripts: { type: 'boolean' }, + error: { type: 'string' }, + summary: { + type: 'object', + additionalProperties: false, + properties: { + critical: { type: 'integer' }, + high: { type: 'integer' }, + medium: { type: 'integer' }, + low: { type: 'integer' }, + unknown: { type: 'integer' }, + total: { type: 'integer' } + } + } + } + }, +``` + +- [ ] **Step 2: Regenerate artefact types** + +Run: `npm run build-types` +Expected: completes; `Artefact` type now has an optional `scan` property. + +- [ ] **Step 3: Add the `ArtefactScan` type and collection to mongo.ts** + +In `api/src/mongo.ts`, add near the `Thumbnail` type: +```ts +export type ScanSeverity = 'critical' | 'high' | 'medium' | 'low' | 'unknown' + +export type ScanFinding = { + id: string + pkgName: string + installedVersion: string + fixedVersion?: string + severity: ScanSeverity + title?: string + primaryUrl?: string +} + +export type ScanLicense = { + pkgName: string + license: string +} + +export type ArtefactScan = { + _id: string // artefact id + scannedAt: string + scannerVersion: string + vulnDbUpdatedAt?: string + vulnerabilities: ScanFinding[] + licenses?: ScanLicense[] +} +``` +Add the collection getter inside the `RegistryMongo` class (next to `thumbnails`): +```ts + get artefactScans () { + return mongoLib.db.collection('artefact-scans') + } +``` +And register an index in `init()` inside the `mongoLib.configure({ ... })` object (next to `thumbnails`): +```ts + 'artefact-scans': { + artefact: [{ _id: 1 }, {}] + }, +``` +> Note: `_id` is already uniquely indexed by Mongo; this entry is a harmless explicit declaration kept for symmetry. If `configure` rejects redefining `_id`, omit the `artefact-scans` entry entirely — no separate index is needed since all queries are by `_id`. + +- [ ] **Step 4: Verify types compile** + +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 5: Commit** + +```bash +git add api/types/artefact/schema.js api/types/artefact/.type api/src/mongo.ts +git commit -m "feat(scanning): add scan summary field, ArtefactScan type and collection" +``` + +--- + +## Task 4: osv-scanner output mapper (pure, TDD) + +**Files:** +- Create: `api/src/scanning/operations.ts` +- Test: `tests/scanning-operations.unit.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Create `tests/scanning-operations.unit.spec.ts`: +```ts +import { test, expect } from '@playwright/test' +import { readFileSync } from 'node:fs' +import { resolve } from 'node:path' +import { mapOsvOutput, severityBucket, summarize } from '../api/src/scanning/operations.ts' + +test.describe('scanning operations', () => { + test('severityBucket maps labels and CVSS scores', () => { + expect(severityBucket('CRITICAL', undefined)).toBe('critical') + expect(severityBucket('HIGH', undefined)).toBe('high') + expect(severityBucket('MODERATE', undefined)).toBe('medium') + expect(severityBucket('LOW', undefined)).toBe('low') + expect(severityBucket(undefined, '9.8')).toBe('critical') + expect(severityBucket(undefined, '7.5')).toBe('high') + expect(severityBucket(undefined, '5.0')).toBe('medium') + expect(severityBucket(undefined, '2.0')).toBe('low') + expect(severityBucket(undefined, undefined)).toBe('unknown') + }) + + test('summarize counts findings by severity', () => { + const summary = summarize([ + { id: 'a', pkgName: 'x', installedVersion: '1', severity: 'high' }, + { id: 'b', pkgName: 'y', installedVersion: '1', severity: 'high' }, + { id: 'c', pkgName: 'z', installedVersion: '1', severity: 'low' } + ] as any) + expect(summary).toEqual({ critical: 0, high: 2, medium: 0, low: 1, unknown: 0, total: 3 }) + }) + + test('mapOsvOutput extracts the bundled minimist vulnerability from the fixture', () => { + const raw = JSON.parse(readFileSync(resolve(import.meta.dirname, 'resources/osv-sample-output.json'), 'utf-8')) + const { vulnerabilities, summary } = mapOsvOutput(raw) + const m = vulnerabilities.find(v => v.pkgName === 'minimist') + expect(m).toBeTruthy() + expect(m!.installedVersion).toBe('0.0.8') + expect(['critical', 'high', 'medium', 'low', 'unknown']).toContain(m!.severity) + expect(typeof m!.id).toBe('string') + expect(summary.total).toBeGreaterThanOrEqual(1) + }) + + test('mapOsvOutput tolerates empty results', () => { + expect(mapOsvOutput({ results: [] }).vulnerabilities).toEqual([]) + expect(mapOsvOutput({}).vulnerabilities).toEqual([]) + }) +}) +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npm run test tests/scanning-operations.unit.spec.ts` +Expected: FAIL — cannot resolve `../api/src/scanning/operations.ts`. + +- [ ] **Step 3: Implement the mapper** + +Create `api/src/scanning/operations.ts`: +```ts +import type { ScanFinding, ScanSeverity, ScanLicense } from '#mongo' + +// --- severity --------------------------------------------------------------- + +const LABEL_TO_BUCKET: Record = { + CRITICAL: 'critical', + HIGH: 'high', + MODERATE: 'medium', + MEDIUM: 'medium', + LOW: 'low' +} + +// osv-scanner exposes severity inconsistently: a textual label in +// vulnerabilities[].database_specific.severity, and/or a numeric CVSS score +// in groups[].max_severity. Prefer the label, fall back to CVSS banding. +export const severityBucket = (label?: string, cvss?: string): ScanSeverity => { + if (label && LABEL_TO_BUCKET[label.toUpperCase()]) return LABEL_TO_BUCKET[label.toUpperCase()] + const score = cvss !== undefined ? Number(cvss) : NaN + if (!Number.isNaN(score)) { + if (score >= 9) return 'critical' + if (score >= 7) return 'high' + if (score >= 4) return 'medium' + if (score > 0) return 'low' + } + return 'unknown' +} + +export type Summary = { critical: number, high: number, medium: number, low: number, unknown: number, total: number } + +export const summarize = (findings: ScanFinding[]): Summary => { + const s: Summary = { critical: 0, high: 0, medium: 0, low: 0, unknown: 0, total: 0 } + for (const f of findings) { s[f.severity]++; s.total++ } + return s +} + +// --- osv json shape (subset we read) --------------------------------------- + +type OsvVuln = { + id: string + summary?: string + database_specific?: { severity?: string } + references?: { type?: string, url?: string }[] + affected?: { package?: { ecosystem?: string }, ranges?: { events?: { fixed?: string }[] }[] }[] +} +type OsvGroup = { ids?: string[], max_severity?: string } +type OsvPackage = { + package: { name: string, version: string, ecosystem?: string } + vulnerabilities?: OsvVuln[] + groups?: OsvGroup[] + licenses?: string[] +} +type OsvOutput = { results?: { packages?: OsvPackage[] }[] } + +const firstFixedVersion = (vuln: OsvVuln): string | undefined => { + for (const aff of vuln.affected ?? []) { + for (const range of aff.ranges ?? []) { + for (const ev of range.events ?? []) { + if (ev.fixed) return ev.fixed + } + } + } + return undefined +} + +const primaryUrl = (vuln: OsvVuln): string | undefined => { + const ref = (vuln.references ?? []).find(r => r.type === 'ADVISORY') ?? (vuln.references ?? [])[0] + return ref?.url +} + +export const mapOsvOutput = (raw: unknown): { vulnerabilities: ScanFinding[], licenses: ScanLicense[], summary: Summary } => { + const out = (raw ?? {}) as OsvOutput + const findings: ScanFinding[] = [] + const licenses: ScanLicense[] = [] + for (const result of out.results ?? []) { + for (const pkg of result.packages ?? []) { + // CVSS score for the package's worst group, used as fallback severity. + const maxCvss = (pkg.groups ?? []) + .map(g => g.max_severity) + .filter((v): v is string => typeof v === 'string') + .sort((a, b) => Number(b) - Number(a))[0] + for (const vuln of pkg.vulnerabilities ?? []) { + findings.push({ + id: vuln.id, + pkgName: pkg.package.name, + installedVersion: pkg.package.version, + fixedVersion: firstFixedVersion(vuln), + severity: severityBucket(vuln.database_specific?.severity, maxCvss), + title: vuln.summary, + primaryUrl: primaryUrl(vuln) + }) + } + for (const lic of pkg.licenses ?? []) { + if (lic && lic !== 'UNKNOWN') licenses.push({ pkgName: pkg.package.name, license: lic }) + } + } + } + return { vulnerabilities: findings, licenses, summary: summarize(findings) } +} +``` +> If the spike (Task 1, Step 5) showed severity living somewhere other than `database_specific.severity` / `groups[].max_severity`, adjust the `OsvVuln`/`OsvGroup` shapes and the two accessor functions to match the captured fixture before running the test. + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npm run test tests/scanning-operations.unit.spec.ts` +Expected: PASS (all 4 tests). + +- [ ] **Step 5: Commit** + +```bash +git add api/src/scanning/operations.ts tests/scanning-operations.unit.spec.ts +git commit -m "feat(scanning): osv-scanner output mapper with unit tests" +``` + +--- + +## Task 5: Install-script detection (pure, TDD) + +**Files:** +- Modify: `api/src/scanning/operations.ts` +- Modify: `tests/scanning-operations.unit.spec.ts` + +- [ ] **Step 1: Write the failing test** + +Append to `tests/scanning-operations.unit.spec.ts` inside the describe block: +```ts + test('detectInstallScripts flags lifecycle install hooks', async () => { + const { detectInstallScripts } = await import('../api/src/scanning/operations.ts') + expect(detectInstallScripts({ scripts: { postinstall: 'node x.js' } })).toBe(true) + expect(detectInstallScripts({ scripts: { preinstall: 'sh y.sh' } })).toBe(true) + expect(detectInstallScripts({ scripts: { install: 'make' } })).toBe(true) + expect(detectInstallScripts({ scripts: { build: 'tsc', test: 'x' } })).toBe(false) + expect(detectInstallScripts({})).toBe(false) + expect(detectInstallScripts(null)).toBe(false) + expect(detectInstallScripts('not an object')).toBe(false) + }) +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npm run test tests/scanning-operations.unit.spec.ts` +Expected: FAIL — `detectInstallScripts` is not exported. + +- [ ] **Step 3: Implement** + +Append to `api/src/scanning/operations.ts`: +```ts +// Whether a (top-level) package.json declares an install lifecycle script. +// Install/preinstall/postinstall run automatically on `npm install` and are a +// primary supply-chain execution vector — surfaced as an advisory flag. +export const detectInstallScripts = (pkgJson: unknown): boolean => { + if (!pkgJson || typeof pkgJson !== 'object') return false + const scripts = (pkgJson as { scripts?: unknown }).scripts + if (!scripts || typeof scripts !== 'object') return false + for (const hook of ['install', 'preinstall', 'postinstall'] as const) { + if (typeof (scripts as Record)[hook] === 'string') return true + } + return false +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npm run test tests/scanning-operations.unit.spec.ts` +Expected: PASS (all 5 tests). + +- [ ] **Step 5: Commit** + +```bash +git add api/src/scanning/operations.ts tests/scanning-operations.unit.spec.ts +git commit -m "feat(scanning): top-level install-script detection" +``` + +--- + +## Task 6: Tarball extraction to a temp dir (TDD) + +**Files:** +- Create: `api/src/scanning/extract.ts` +- Test: `tests/scanning-extract.unit.spec.ts` +- Modify: `api/package.json` (add `tar` dependency) + +- [ ] **Step 1: Add node-tar** + +Run: `npm install -w api tar@7` +Expected: `tar` appears in `api/package.json` dependencies. + +- [ ] **Step 2: Write the failing test** + +Create `tests/scanning-extract.unit.spec.ts`: +```ts +import { test, expect } from '@playwright/test' +import { Readable } from 'node:stream' +import { mkdtemp, rm, readFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { createTestTarball } from './support/test-tarball.ts' +import { extractTarballToDir } from '../api/src/scanning/extract.ts' + +test.describe('scanning extract', () => { + test('extracts a gzipped tarball to disk preserving the package/ layout', async () => { + const buf = await createTestTarball({ + name: '@test/pkg', + version: '1.0.0', + extraEntries: [{ name: 'package/node_modules/foo/package.json', content: '{"name":"foo","version":"1.2.3"}' }] + }) + const dir = await mkdtemp(join(tmpdir(), 'scan-test-')) + try { + await extractTarballToDir(Readable.from(buf), dir, { maxEntries: 1000 }) + const top = JSON.parse(await readFile(join(dir, 'package', 'package.json'), 'utf-8')) + expect(top.name).toBe('@test/pkg') + const dep = JSON.parse(await readFile(join(dir, 'package', 'node_modules', 'foo', 'package.json'), 'utf-8')) + expect(dep.version).toBe('1.2.3') + } finally { + await rm(dir, { recursive: true, force: true }) + } + }) + + test('rejects a tarball exceeding the entry cap', async () => { + const extraEntries = Array.from({ length: 50 }, (_, i) => ({ name: `package/f${i}.txt`, content: 'x' })) + const buf = await createTestTarball({ name: '@test/big', version: '1.0.0', extraEntries }) + const dir = await mkdtemp(join(tmpdir(), 'scan-test-')) + try { + await expect(extractTarballToDir(Readable.from(buf), dir, { maxEntries: 10 })).rejects.toThrow(/entries/) + } finally { + await rm(dir, { recursive: true, force: true }) + } + }) +}) +``` + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `npm run test tests/scanning-extract.unit.spec.ts` +Expected: FAIL — cannot resolve `extract.ts`. + +- [ ] **Step 4: Implement** + +Create `api/src/scanning/extract.ts`: +```ts +import type { Readable } from 'node:stream' +import { pipeline } from 'node:stream/promises' +import * as tar from 'tar' +import { httpError } from '@data-fair/lib-utils/http-errors.js' + +export type ExtractOpts = { maxEntries: number } + +// Extract a gzipped npm tarball stream into `dir`. node-tar handles gunzip and +// guards against path traversal (it strips `..` and refuses absolute paths by +// default). An entry counter caps the file count to mirror upload-time limits. +export const extractTarballToDir = async (stream: Readable, dir: string, opts: ExtractOpts): Promise => { + let count = 0 + const extract = tar.x({ + cwd: dir, + gzip: true, + // Reject anything outside cwd; node-tar already strips '..' but be explicit. + filter: (path: string) => { + count++ + if (count > opts.maxEntries) throw httpError(413, `tarball exceeds ${opts.maxEntries} entries`) + return !path.includes('..') + } + }) + await pipeline(stream, extract) +} +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `npm run test tests/scanning-extract.unit.spec.ts` +Expected: PASS (both tests). + +- [ ] **Step 6: Commit** + +```bash +git add api/package.json package-lock.json api/src/scanning/extract.ts tests/scanning-extract.unit.spec.ts +git commit -m "feat(scanning): tarball-to-dir extraction with entry cap" +``` + +--- + +## Task 7: osv-scanner runner + Scanner interface + +**Files:** +- Create: `api/src/scanning/runner.ts` + +This task has no in-process unit test (it spawns an external binary). It is covered by the gated integration check in Step 3 and exercised end-to-end in Task 11 verification. The pure parsing it delegates to is already tested (Task 4). + +- [ ] **Step 1: Implement the runner** + +Create `api/src/scanning/runner.ts`: +```ts +import { spawn } from 'node:child_process' +import { readFile, writeFile, mkdtemp, rm, mkdir } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import config from '#config' +import type { ScanFinding, ScanLicense } from '#mongo' +import { mapOsvOutput, summarize, detectInstallScripts, type Summary } from './operations.ts' + +export type ScanResult = { + vulnerabilities: ScanFinding[] + licenses: ScanLicense[] + summary: Summary + hasInstallScripts: boolean + scannerVersion: string +} + +export interface Scanner { + // Scan an already-extracted artefact directory. + scanDir (dir: string): Promise + // Refresh the local offline DB (online). Folded into the periodic job. + refreshDb (): Promise + version (): Promise +} + +const run = (args: string[], timeoutMs: number): Promise<{ code: number, stdout: string, stderr: string }> => + new Promise((resolve, reject) => { + const child = spawn(config.scanning.osvScannerPath, args, { timeout: timeoutMs }) + let stdout = '' + let stderr = '' + child.stdout.on('data', d => { stdout += d }) + child.stderr.on('data', d => { stderr += d }) + child.on('error', reject) + child.on('close', (code) => resolve({ code: code ?? -1, stdout, stderr })) + }) + +// osv-scanner exits 0 = no vulns, 1 = vulns found, >1 = real error +// (e.g. 128 = "no package sources found"). Flags below pinned by the Task 1 +// spike against osv-scanner v2.2.3. +const EXIT_VULNS_FOUND = 1 + +// The scalibr package.json extractor is NOT enabled by default; without it a +// directory scan of bundled (lockfile-less) node_modules finds nothing. +const PLUGIN_ARGS = ['--experimental-plugins', 'javascript/packagejson'] + +class OsvScanner implements Scanner { + async version (): Promise { + const { stdout } = await run(['--version'], 30_000) + // First line: "osv-scanner version: X.Y.Z" + return stdout.trim().split('\n')[0] || 'unknown' + } + + async refreshDb (): Promise { + // The DB is only downloaded when >=1 package is found, so scan a tiny dummy + // dir holding one package.json (with --allow-no-lockfiles). + await mkdir(config.scanning.dbDir, { recursive: true }) + const dummy = await mkdtemp(join(tmpdir(), 'osv-db-refresh-')) + try { + await mkdir(join(dummy, 'node_modules', 'left-pad'), { recursive: true }) + await writeFile(join(dummy, 'node_modules', 'left-pad', 'package.json'), '{"name":"left-pad","version":"1.0.0"}') + const args = [ + 'scan', 'source', '--recursive', + ...PLUGIN_ARGS, + '--offline-vulnerabilities', + '--download-offline-databases', + '--local-db-path', config.scanning.dbDir, + '--allow-no-lockfiles', + '--format', 'json', + dummy + ] + const { code, stderr } = await run(args, config.scanning.timeoutSeconds * 1000) + if (code > EXIT_VULNS_FOUND) throw new Error(`osv-scanner db refresh failed (exit ${code}): ${stderr.slice(0, 500)}`) + } finally { + await rm(dummy, { recursive: true, force: true }).catch(() => {}) + } + } + + async scanDir (dir: string): Promise { + const args = [ + 'scan', 'source', '--recursive', + ...PLUGIN_ARGS, + '--offline-vulnerabilities', + '--local-db-path', config.scanning.dbDir, + '--allow-no-lockfiles', + '--format', 'json', + dir + ] + const { code, stdout, stderr } = await run(args, config.scanning.timeoutSeconds * 1000) + if (code > EXIT_VULNS_FOUND) { + throw new Error(`osv-scanner failed (exit ${code}): ${stderr.slice(0, 500)}`) + } + const raw = stdout.trim() ? JSON.parse(stdout) : { results: [] } + const { vulnerabilities, licenses, summary } = mapOsvOutput(raw) + + let hasInstallScripts = false + try { + const topPkg = JSON.parse(await readFile(join(dir, 'package', 'package.json'), 'utf-8')) + hasInstallScripts = detectInstallScripts(topPkg) + } catch { /* top-level package.json missing/unreadable — leave false */ } + + return { vulnerabilities, licenses, summary, hasInstallScripts, scannerVersion: await this.version() } + } +} + +export { summarize } +export const osvScanner: Scanner = new OsvScanner() +``` + +- [ ] **Step 2: Verify types compile** + +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 3: Manual gated integration check (skipped if binary absent)** + +Run (uses the spike binary + DB from Task 1): +```bash +WORK=$(mktemp -d); mkdir -p "$WORK/extracted/package/node_modules/minimist" +echo '{"name":"@test/p","version":"1.0.0","scripts":{"postinstall":"node x.js"}}' > "$WORK/extracted/package/package.json" +echo '{"name":"minimist","version":"0.0.8"}' > "$WORK/extracted/package/node_modules/minimist/package.json" +node --experimental-strip-types -e " +import('./api/src/scanning/runner.ts').then(async ({osvScanner}) => { + process.env.NODE_CONFIG = JSON.stringify({scanning:{osvScannerPath:'/tmp/osv-scanner',dbDir:'/tmp/osv-db',timeoutSeconds:300,enabled:true}}); + const r = await osvScanner.scanDir('$WORK/extracted'); + console.log(JSON.stringify({summary:r.summary, hasInstallScripts:r.hasInstallScripts, sample:r.vulnerabilities[0]}, null, 2)); +})" 2>&1 | tail -20 +``` +Expected: prints a summary with `total >= 1`, `hasInstallScripts: true`, and a sample finding for `minimist`. (If config injection via env is awkward, this check can instead be performed during Task 11 end-to-end verification against the dev server.) + +- [ ] **Step 4: Commit** + +```bash +git add api/src/scanning/runner.ts +git commit -m "feat(scanning): osv-scanner subprocess runner behind Scanner interface" +``` + +--- + +## Task 8: Scanning service (orchestration) + +**Files:** +- Create: `api/src/scanning/service.ts` + +Orchestration depends on Mongo + filesystem + the binary, so it is verified through the API tests (Task 9) and end-to-end (Task 11) rather than an in-process unit test. The `Scanner` is injectable so a fake can be used in any future in-process test. + +- [ ] **Step 1: Implement the service** + +Create `api/src/scanning/service.ts`: +```ts +import { mkdtemp, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import locks from '@data-fair/lib-node/locks.js' +import { internalError } from '@data-fair/lib-node/observer.js' +import mongo from '#mongo' +import config from '#config' +import { filesStorage } from '../files-storage/index.ts' +import { extractTarballToDir } from './extract.ts' +import { osvScanner, type Scanner } from './runner.ts' + +let scanner: Scanner = osvScanner +// Test seam: swap the scanner implementation (no production caller). +export const __setScanner = (s: Scanner) => { scanner = s } + +// In-process concurrency gate. +let active = 0 +const waiters: (() => void)[] = [] +const acquireSlot = async () => { + if (active >= config.scanning.concurrency) { + await new Promise(res => waiters.push(res)) + } + active++ +} +const releaseSlot = () => { + active-- + const next = waiters.shift() + if (next) next() +} + +const setStatus = (id: string, scan: Record) => + mongo.artefacts.updateOne({ _id: id }, { $set: { scan } }) + +// Mark an artefact pending and kick a background scan. Never throws to the +// caller; failures are recorded on the doc. +export const enqueueScan = async (id: string): Promise => { + if (!config.scanning.enabled) return + await setStatus(id, { status: 'pending', queuedAt: new Date().toISOString() }) + // Fire-and-forget; do not block the upload response. + runScanNow(id).catch(err => internalError('scan', err)) +} + +// Run a full scan synchronously (used by the background task + rescanAll). +export const runScanNow = async (id: string, opts: { refreshDb?: boolean } = {}): Promise => { + if (!config.scanning.enabled) return + const lockId = `scan-${id}` + if (!await locks.acquire(lockId)) return // another instance/worker has it + await acquireSlot() + let dir: string | undefined + try { + const artefact = await mongo.artefacts.findOne({ _id: id }) + if (!artefact || artefact.format !== 'npm' || !artefact.path) return + + await setStatus(id, { ...artefact.scan, status: 'running', startedAt: new Date().toISOString() }) + if (opts.refreshDb) await scanner.refreshDb() + + const { body } = await filesStorage.readStream(artefact.path) + dir = await mkdtemp(join(tmpdir(), 'osv-scan-')) + await extractTarballToDir(body, dir, { maxEntries: config.maxTarEntries }) + + const result = await scanner.scanDir(dir) + const now = new Date().toISOString() + + await mongo.artefactScans.replaceOne( + { _id: id }, + { + _id: id, + scannedAt: now, + scannerVersion: result.scannerVersion, + vulnerabilities: result.vulnerabilities, + ...(result.licenses.length ? { licenses: result.licenses } : {}) + }, + { upsert: true } + ) + await setStatus(id, { + status: 'success', + finishedAt: now, + scannerVersion: result.scannerVersion, + hasInstallScripts: result.hasInstallScripts, + summary: result.summary + }) + } catch (err) { + await setStatus(id, { status: 'error', finishedAt: new Date().toISOString(), error: (err as Error).message?.slice(0, 500) }) + .catch(e => internalError('scan', e)) + } finally { + if (dir) await rm(dir, { recursive: true, force: true }).catch(() => {}) + releaseSlot() + await locks.release(lockId) + } +} + +// Periodic job: refresh the DB once, then rescan every npm artefact. +export const rescanAll = async (): Promise => { + if (!config.scanning.enabled) return + try { + await scanner.refreshDb() + } catch (err) { + internalError('scan-db-refresh', err) + } + const ids = await mongo.artefacts.find({ format: 'npm' }, { projection: { _id: 1 } }).toArray() + for (const { _id } of ids) { + await runScanNow(_id).catch(err => internalError('scan', err)) + } +} +``` + +- [ ] **Step 2: Verify types compile** + +Run: `npm run check-types` +Expected: PASS. + +- [ ] **Step 3: Commit** + +```bash +git add api/src/scanning/service.ts +git commit -m "feat(scanning): scan orchestration service (enqueue, run, rescanAll)" +``` + +--- + +## Task 9: API endpoints + admin-only stripping + test-env injection + +**Files:** +- Create: `api/src/scanning/router.ts` +- Modify: `api/src/artefacts/router.ts` +- Modify: `api/src/app.ts` +- Test: `tests/scanning.api.spec.ts` + +- [ ] **Step 1: Write the failing API test** + +Create `tests/scanning.api.spec.ts`: +```ts +import { test, expect } from '@playwright/test' +import FormData from 'form-data' +import { superAdmin, axiosWithApiKey, anonymousAx, axiosAuth, clean } from './support/axios.ts' +import { createTestTarball } from './support/test-tarball.ts' + +const id = '@test/pkg@1' +const enc = encodeURIComponent(id) + +const injectScan = async (artefactId: string, summary: any, findings: any[]) => { + await anonymousAx.put( + `http://localhost:${process.env.DEV_API_PORT}/api/test-env/artefacts/${encodeURIComponent(artefactId)}/scan`, + { summary, findings } + ) +} + +test.describe('Scanning', () => { + let uploadApiKey: string + test.beforeEach(async () => { + await clean() + const admin = await superAdmin + const keyRes = await admin.post('/api/v1/api-keys', { type: 'upload', name: 'test-upload' }) + uploadApiKey = keyRes.data.key + const form = new FormData() + form.append('file', await createTestTarball({ name: '@test/pkg', version: '1.0.0' }), { filename: 'p.tgz', contentType: 'application/gzip' }) + await axiosWithApiKey(uploadApiKey).post('/api/v1/artefacts/npm/' + enc, form, { headers: form.getHeaders() }) + await admin.patch('/api/v1/artefacts/' + enc, { public: true }) + }) + + test('scan summary is visible to admins on GET /:id but stripped for others', async () => { + await injectScan(id, { critical: 0, high: 1, medium: 0, low: 0, unknown: 0, total: 1 }, + [{ id: 'GHSA-x', pkgName: 'minimist', installedVersion: '0.0.8', severity: 'high' }]) + + const admin = await superAdmin + const adminRes = await admin.get('/api/v1/artefacts/' + enc) + expect(adminRes.data.scan?.summary?.high).toBe(1) + + // Public/consumer view must NOT include scan data. + const userAx = await axiosAuth('test1-user1') + const userRes = await userAx.get('/api/v1/artefacts/' + enc) + expect(userRes.data.scan).toBeUndefined() + }) + + test('GET /:id/scan returns full findings for admin, 403/401 for non-admin', async () => { + await injectScan(id, { critical: 0, high: 1, medium: 0, low: 0, unknown: 0, total: 1 }, + [{ id: 'GHSA-x', pkgName: 'minimist', installedVersion: '0.0.8', fixedVersion: '0.2.1', severity: 'high' }]) + + const admin = await superAdmin + const res = await admin.get('/api/v1/artefacts/' + enc + '/scan') + expect(res.data.vulnerabilities[0].pkgName).toBe('minimist') + + try { + await anonymousAx.get('/api/v1/artefacts/' + enc + '/scan') + expect(true).toBe(false) + } catch (err: any) { + expect([401, 403]).toContain(err.status) + } + }) + + test('GET /:id/scan returns 404 when never scanned', async () => { + const admin = await superAdmin + try { + await admin.get('/api/v1/artefacts/' + enc + '/scan') + expect(true).toBe(false) + } catch (err: any) { + expect(err.status).toBe(404) + } + }) + + test('POST /:id/scan returns 503 when scanning is disabled (dev default)', async () => { + const admin = await superAdmin + try { + const res = await admin.post('/api/v1/artefacts/' + enc + '/scan') + // If scanning is enabled in this env it returns 202 instead. + expect([202, 503]).toContain(res.status) + } catch (err: any) { + expect([503]).toContain(err.status) + } + }) + + test('POST /:id/scan rejects a non-admin caller', async () => { + try { + await anonymousAx.post('/api/v1/artefacts/' + enc + '/scan') + expect(true).toBe(false) + } catch (err: any) { + expect([401, 403]).toContain(err.status) + } + }) +}) +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npm run test tests/scanning.api.spec.ts` +Expected: FAIL — endpoints + test-env helper not present (404s). + +- [ ] **Step 3: Implement the scanning sub-router** + +Create `api/src/scanning/router.ts`: +```ts +import { Router } from 'express' +import { session } from '@data-fair/lib-express/index.js' +import { httpError } from '@data-fair/lib-utils/http-errors.js' +import mongo from '#mongo' +import config from '#config' +import { getArtefactById } from '../artefacts/service.ts' +import { enqueueScan } from './service.ts' + +// Mounted at /api/v1/artefacts/:id/scan (mergeParams to read :id). +const router = Router({ mergeParams: true }) +export default router + +// Full findings (admin only). +router.get('/', async (req, res, next) => { + try { + await session.reqAdminMode(req) + const scan = await mongo.artefactScans.findOne({ _id: req.params.id }) + if (!scan) throw httpError(404, 'no scan for this artefact') + res.json(scan) + } catch (err) { next(err) } +}) + +// Trigger an on-demand (re)scan (admin only). +router.post('/', async (req, res, next) => { + try { + await session.reqAdminMode(req) + if (!config.scanning.enabled) throw httpError(503, 'scanning is not enabled on this deployment') + const artefact = await getArtefactById(req.params.id) + if (!artefact) throw httpError(404, 'artefact not found') + if (artefact.format !== 'npm') throw httpError(400, 'only npm artefacts can be scanned') + await enqueueScan(req.params.id) + res.status(202).json({ status: 'pending' }) + } catch (err) { next(err) } +}) +``` + +- [ ] **Step 4: Mount the sub-router and strip `scan` for non-admins** + +In `api/src/artefacts/router.ts`: + +Add the import near the other router imports: +```ts +import scanRouter from '../scanning/router.ts' +``` +Mount it right after the thumbnail sub-router mount: +```ts +router.use('/:id/scan', scanRouter) +``` +Add a stripping helper after `resolveCaller`/imports (top-level in the file): +```ts +import type { Caller } from '../access.ts' +const stripScan = (artefact: T, caller: Caller): T => { + if (!caller.admin && artefact.scan !== undefined) { + const { scan, ...rest } = artefact + return rest as T + } + return artefact +} +``` +In the `GET /` list handler, change the response to strip per-result. Replace: +```ts + const { results, count } = await listArtefacts(filter, { sort, skip, size }) + res.json({ results, count }) +``` +with: +```ts + const caller = await resolveCaller(req) + const { results, count } = await listArtefacts(filter, { sort, skip, size }) + res.json({ results: results.map(r => stripScan(r, caller)), count }) +``` +> Note: `filter` in this handler is currently `artefactAccessFilter(await resolveCaller(req))` inline. Refactor to resolve the caller once: `const caller = await resolveCaller(req); const filter = artefactAccessFilter(caller)`, then reuse `caller` for `stripScan`. + +In the `GET /:id` detail handler, strip before responding. Replace: +```ts + const filter = artefactAccessFilter(await resolveCaller(req)) + const artefact = await getArtefact(req.params.id, filter) + if (!artefact) throw httpError(404, 'artefact not found') + res.json(artefact) +``` +with: +```ts + const caller = await resolveCaller(req) + const artefact = await getArtefact(req.params.id, artefactAccessFilter(caller)) + if (!artefact) throw httpError(404, 'artefact not found') + res.json(stripScan(artefact, caller)) +``` + +- [ ] **Step 5: Add the test-env scan-injection helper** + +In `api/src/app.ts`, the test-env routes live **inside the `if (process.env.NODE_ENV === 'development') { ... }` block** (which already imports `mongo` and uses `assertReqInternal`; the global `express.json()` is already applied, so no per-route body parser is needed). Add a new route inside that block, after the `PUT .../origin` helper: +```ts + app.put('/api/test-env/artefacts/:id/scan', async (req, res) => { + assertReqInternal(req) + const id = decodeURIComponent(req.params.id) + const { summary, findings } = req.body + await mongo.artefacts.updateOne( + { _id: id }, + { $set: { scan: { status: 'success', finishedAt: new Date().toISOString(), summary } } } + ) + await mongo.artefactScans.replaceOne( + { _id: id }, + { _id: id, scannedAt: new Date().toISOString(), scannerVersion: 'test', vulnerabilities: findings ?? [] }, + { upsert: true } + ) + res.send() + }) +``` +Also add scan cleanup to the existing `DELETE /api/test-env` handler, alongside the other `deleteMany` calls: +```ts + await mongo.artefactScans.deleteMany({}) +``` +> The test posts to this route via `anonymousAx` (the internal-origin axios used by `clean()` / `setArtefactOrigin`), so `assertReqInternal` passes exactly as it does for the existing helpers. + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `npm run test tests/scanning.api.spec.ts` +Expected: PASS (all 5 tests). + +- [ ] **Step 7: Run the existing artefacts tests to confirm no regression in list/detail** + +Run: `npm run test tests/artefacts.api.spec.ts` +Expected: PASS (the stripping change must not alter existing responses, since no scan field is set in those tests). + +- [ ] **Step 8: Commit** + +```bash +git add api/src/scanning/router.ts api/src/artefacts/router.ts api/src/app.ts tests/scanning.api.spec.ts +git commit -m "feat(scanning): scan endpoints, admin-only stripping, test-env helper" +``` + +--- + +## Task 10: Wire enqueue-on-upload and the periodic rescan + +**Files:** +- Modify: `api/src/artefacts/service.ts` +- Modify: `api/src/server.ts` + +- [ ] **Step 1: Enqueue a scan after a successful npm upload** + +In `api/src/artefacts/service.ts`, add the import at the top (after the existing imports): +```ts +import { enqueueScan } from '../scanning/service.ts' +``` +At the end of `commitNpmUpload`, just before `return (await mongo.artefacts.findOne({ _id: id }))!`, add: +```ts + // Advisory scan runs in the background; never blocks the upload response. + await enqueueScan(id) +``` +> `enqueueScan` is a no-op when `config.scanning.enabled` is false and never throws, so this is safe in all environments. + +- [ ] **Step 2: Add the periodic rescan + boot DB warm-up in server.ts** + +In `api/src/server.ts`, add the import: +```ts +import { rescanAll } from './scanning/service.ts' +``` +Add a timer handle near `let syncTimer`: +```ts +let rescanTimer: ReturnType | undefined +``` +In `start()`, after the existing daily-sync `setInterval` block, add: +```ts + if (config.scanning.enabled) { + // Warm the DB + do an initial pass shortly after boot, then on the interval. + rescanAll().catch(err => internalError('rescan-all', err)) + rescanTimer = setInterval(() => { + rescanAll().catch(err => internalError('rescan-all', err)) + }, config.scanning.rescanIntervalHours * 60 * 60 * 1000) + } +``` +In `stop()`, alongside `if (syncTimer) clearInterval(syncTimer)`, add: +```ts + if (rescanTimer) clearInterval(rescanTimer) +``` + +- [ ] **Step 3: Verify types compile and existing tests still pass** + +Run: `npm run check-types && npm run test tests/artefacts.api.spec.ts` +Expected: PASS (uploads still succeed; enqueue is a no-op with scanning disabled). + +- [ ] **Step 4: Commit** + +```bash +git add api/src/artefacts/service.ts api/src/server.ts +git commit -m "feat(scanning): scan on upload + periodic rescan wiring" +``` + +--- + +## Task 11: End-to-end verification with osv-scanner enabled (manual) + +This proves the whole pipeline against the real binary. It requires temporarily enabling scanning in the dev API config. **Per AGENTS.md the agent must not start/stop dev processes — ask the user to apply the config and restart the API, or run a one-off API instance.** + +- [ ] **Step 1: Make the osv-scanner binary available on the dev host** + +Ensure `/tmp/osv-scanner` from Task 1 exists (or install to a PATH location). Note its path. + +- [ ] **Step 2: Ask the user to enable scanning for the dev API** + +Provide the user these env vars (or a `config/development.js` override) and ask them to restart the dev API: +``` +SCANNING_ENABLED=true +OSV_SCANNER_PATH=/tmp/osv-scanner +OSV_DB_DIR=/tmp/osv-db +``` +Confirm via `bash dev/status.sh` that the API is running after their restart. + +- [ ] **Step 3: Upload a vulnerable bundled-deps tarball and watch the scan complete** + +Run (uses the dev ports; adjust auth to your superadmin flow or an upload API key): +```bash +# Build the vulnerable tarball +WORK=$(mktemp -d); mkdir -p "$WORK/package/node_modules/minimist" +echo '{"name":"@test/vuln","version":"1.0.0","scripts":{"postinstall":"node x.js"}}' > "$WORK/package/package.json" +echo '{"name":"minimist","version":"0.0.8"}' > "$WORK/package/node_modules/minimist/package.json" +tar -C "$WORK" -czf "$WORK/v.tgz" package +# Upload via an upload API key (create one as superadmin first), then poll: +# (pseudo — use your test helper or curl with x-api-key) +``` +Then poll the artefact as admin until `scan.status` is `success`: +```bash +# curl -s -H 'Cookie: ' "http://localhost:$DEV_API_PORT/api/v1/artefacts/$(python3 -c 'import urllib.parse;print(urllib.parse.quote("@test/vuln@1"))')" | jq .scan +``` +Expected: `scan.status: "success"`, `scan.summary.total >= 1`, `scan.hasInstallScripts: true`. And `GET /:id/scan` returns a `minimist` finding. + +- [ ] **Step 4: Record the result** + +Note the observed summary in the PR description / commit message. No code commit in this task unless the spike-pinned flags in `runner.ts` needed correction — if so: +```bash +git add api/src/scanning/runner.ts +git commit -m "fix(scanning): correct osv-scanner flags per e2e verification" +``` + +--- + +## Task 12: Admin UI — scan panel + +**Files:** +- Modify: `ui/src/components/artefact-admin.vue` +- Modify: the i18n locale file(s) backing that component (locate via existing `t('...')` usage in the file) + +- [ ] **Step 1: Add a scan card to the admin component** + +In `ui/src/components/artefact-admin.vue`, add a new `` in the template (after the editable-metadata card), shown only for npm artefacts. It reads `artefact.scan` (already on the doc for admins) and lazily fetches full findings: +```vue + + + + {{ t('scanTitle') }} + + + {{ t('rescan') }} + + + +
{{ t('scanNever') }}
+ + + +``` + +- [ ] **Step 2: Add the script logic** + +In the ` +``` + +Notes for the implementer: +- `useFetch`, `useLocaleDayjs` are auto-imported composables already used elsewhere (`ui/src/pages/index.vue` uses both). `dayjs(...).fromNow()` is used to render the oldest-scan age; if `fromNow` is not available on the configured dayjs, fall back to `dayjs(health.oldestScanAt).format('L')`. +- `SEVERITY_ORDER` is imported explicitly here for clarity; it is also auto-importable. Match whichever the linter prefers (auto-import means you can drop the `import { SEVERITY_ORDER }` line). + +- [ ] **Step 2: Mount the section in the admin dashboard** + +In `ui/src/pages/admin/index.vue`, add a `section-tabs` block (e.g. as the first section, before API keys) and import the component. + +In the template, add: +```vue + + + +``` + +In ` diff --git a/ui/src/components/artefact-admin.vue b/ui/src/components/artefact-admin.vue index adee8c7..abccc08 100644 --- a/ui/src/components/artefact-admin.vue +++ b/ui/src/components/artefact-admin.vue @@ -95,6 +95,114 @@ + + + + {{ t('scanTitle') }} + + + {{ t('rescan') }} + + + +
+ {{ t('scanNever') }} +
+ + + + diff --git a/ui/src/pages/admin/index.vue b/ui/src/pages/admin/index.vue index 582618f..5768836 100644 --- a/ui/src/pages/admin/index.vue +++ b/ui/src/pages/admin/index.vue @@ -1,5 +1,16 @@