WSS-727 Do not cache SwA attachment References when verifying a Signature

[shunkica]

JIRA: https://issues.apache.org/jira/browse/WSS-727

The Problem

When you verify a SOAP message with a signed SwA attachment, it needs about 2.3x the attachment's size in heap memory. This happens even if the attachment is already on disk or if you provide a streaming source. Compare that to signing the same message, which only needs ~20 MB consistently. That's a huge difference.

The culprit is a single flag in SignatureProcessor.verifyXMLSignature():

XMLValidateContext context = new DOMValidateContext(key, elem);
context.setProperty("javax.xml.crypto.dsig.cacheReference", Boolean.TRUE);

This flag tells Santuario to cache stuff, but it's a blunt instrument. In DOMReference.transform() (xmlsec 4.0.4), it flips on two separate caches that have nothing to do with each other:

Boolean cache = (Boolean) context.getProperty("javax.xml.crypto.dsig.cacheReference");
if (cache != null && cache) {
    this.derefData = copyDerefData(dereferencedData);
    dos = new DigesterOutputStream(md, true);   // <-- keeps every byte fed to the digest
} else {
    dos = new DigesterOutputStream(md);
}

The two caches:

1. Dereferenced data - cheap, and WSS4J actually needs this. It's used by buildProtectedRefs() to figure out what each Reference covered.
2. Pre-digested input - the problem. The DigesterOutputStream buffers it all in an UnsyncByteArrayOutputStream, which doubles in size as it grows, then copies the whole thing again in getInputStream().

For attachments, AttachmentContentSignatureTransform feeds the attachment directly into that stream. So a 200 MB attachment ends up consuming a 256 MB backing array plus a 200 MB copy - ~456 MB live at once. This buffered copy is useless: the only thing that reads it is Reference.getDigestInputStream(), and WSS4J never calls that.

java.lang.OutOfMemoryError: Java heap space
    at org.apache.xml.security.utils.UnsyncByteArrayOutputStream.expandSize(UnsyncByteArrayOutputStream.java:113)
    at org.apache.jcp.xml.dsig.internal.DigesterOutputStream.write(DigesterOutputStream.java:83)
    at org.apache.wss4j.dom.transform.AttachmentContentSignatureTransform.processAttachment(AttachmentContentSignatureTransform.java:218)
    ...

The Fix

The solution has two parts:

1. Replace xmlSignature.validate(context) with an explicit loop. Instead of one call, we do:

   • Check the SignatureValue
   • Validate each Reference individually
   • For attachment References (identified by their Transform algorithm), toggle off cacheReference

2. Update buildProtectedRefs() to recognize attachments by their Transform algorithm. This is essential. With caching off, getDereferencedData() returns null, so toggling the flag alone would break every signed attachment with FAILED_CHECK. The fix keeps buildProtectedRefs() working: it still produces the same WSDataRef with the synthesised <attachment> element and setAttachment(true).

Why this preserves the existing behavior:

• The flag is read per-Reference at transform time, so we can toggle it between Reference.validate() calls without issues.

• Reference.validate() caches its result, so when debug logging re-validates a Reference we already checked, it just uses the cached status, it doesn't re-transform. References the loop hasn't reached yet get validated there, and the caching rule applies to them too. This matters more than it sounds: if the SignatureValue check fails (a common failure), we bail out before validating any References. Without this, turning on debug logging to diagnose the failure would buffer every signed attachment on that message, leading to the very OutOfMemoryError we're fixing. The old code did exactly this.

• Non-attachment References still cache, so element recovery, the STR-dereference path (WSS-222), and the anti-wrapping checks all keep working. We restore the flag to TRUE when we're done.

• The explicit loop is fully equivalent to DOMXMLSignature.validate(): Manifest validation only runs if org.jcp.xml.dsig.validateManifests is set (WSS4J never sets it), and the XMLSignature's cached validation status is never read because the object doesn't escape SignatureProcessor.

• The short-circuit semantics match the old behavior - still bail on first failure.

• ws-security-stax doesn't use this property, so no impact there.

Testing

A new test, AttachmentTest.testXMLAttachmentContentSignatureDataRef, verifies that the WSDataRef for a signed attachment still has isAttachment() set and still carries the synthesised SwA attachment element. This ensures the buildProtectedRefs() change does what it's supposed to.

Memory measurements

200 MB attachment from the WSS-727 repro; source stream is mark/reset-capable so it contributes no heap

Scenario	Heap limit	Result
Sign 200 MB attachment	256m	OK, peak 17 MB
Verify 200 MB attachment (before fix)	256m	OutOfMemoryError
Verify 200 MB attachment (before fix)	512m	OK, peak 466 MB
Verify 200 MB attachment (after fix)	256m	OK, peak 22 MB

Notes

History: The cacheReference flag has been there since the WSS4J 2.0 rename (commit f647a91), so the bug affects 2.x and 3.x as well. Tested on 4.0.1 and current master.

The bigger picture: The digest input buffer also costs ~3.2x the signed content for regular element References (minimum -Xmx: 160m vs 96m for a 20 MB Body, or 288m vs 160m for 40 MB). The key difference is asymptotic: element content is already in the DOM, so the cache is a constant-factor overhead you can outrun by raising heap; attachment content is streaming and otherwise never in memory, so the cache turns constant-heap streaming into O(attachment-size) that no fixed heap covers. This fix is narrowly scoped to SwA attachments - the only case with no caller-side workaround.

For element and STR References, the dereferenced data is the authoritative record of what was digested and is essential for WSDataRef and the anti-wrapping checks. You can't just disable caching for those without losing that data. A proper fix for all Reference types would require Santuario to expose separate properties for the two caches instead of one toggle for both - that would let buildProtectedRefs() fetch the dereferenced data without buffering the digest input. That's a design change upstream that would need a new Santuario release.

Related: WSS-638 is about processAttachment() buffering non-mark-capable source streams via BufferedInputStream.mark(Integer.MAX_VALUE). That one still happens on both sign and verify, and it's not fixed here. The workaround is to supply a disk-backed, mark-capable stream. The bug we're fixing had no workaround because the digest-side cache retained the attachment regardless of how the stream behaved.