[einsz]

Description

A Sound with no data plays what its sampleData listeners write into SampleDataEvent.data. Ruffle never dispatched the event, so play() queued forever, and synths, trackers and engines that mix their own audio were silent.

play() on such a sound now starts a channel that asks the listeners for samples as it plays. The samples go into a SampleQueue shared with the audio backend, which a new mixer stream plays as they arrive. The stream plays silence while it waits, and counts only the samples it played, so SoundChannel.position works through the existing get_sound_position. AudioBackend gets one new method, start_sample_queue, implemented in the mixer macro and the null backend.

I checked what Flash Player 32 does with probe SWFs in the release projector, with its audio going to a PulseAudio null sink, which runs in real time:

• The first sampleData event is dispatched inside play(), at position 0. Flash keeps asking while the listeners write blocks of 8192 samples, up to five, before play() returns.
• After that it asks again as its buffer drains, about once per 2048 samples played. position counts the samples generated so far on that channel. Every channel of a sound asks separately, starting from 0.
• Writing fewer than 2048 samples (including none) ends the sound once they have played, and soundComplete reports channel.position at the end of the samples. Writing more than 8192 plays none of them and ends the sound.
• The samples counted are those before data.position, not data.length: data is the same ByteArray for every request of a sound, rewound to 0 but not cleared.
• Floats are read with the ByteArray's endianness; a little-endian sine plays the same as a big-endian one.
• play() on a sound with no data and no sampleData listener throws ArgumentError SWF calculator does not work in 2020-12-22 #2068. A sound that has called load() never asks its listeners.

Flash asks from its audio thread. Ruffle asks once per Player::tick, after the frames have run, keeping the queue ahead of playback by 4096 samples plus two ticks, or two mixing buffers on backends that report one (position_resolution, i.e. web), capped at a second. That is more latency than Flash's, but avoids gaps between ticks.

Testing

Two new tests, with output checked against Flash Player 32:

• avm2/sound_sampledata: event positions, the reused ByteArray, ending on short, empty, rewound and oversized blocks, replaying, several channels, error SWF calculator does not work in 2020-12-22 #2068, and load().
• avm2/sound_sampledata_audio: audio assertions for little-endian samples and a SoundTransform. Recorded in Flash, the same SWF peaks at the same levels.

Both use num_ticks: with num_frames the harness calls run_frame without Player::tick, so nothing asks for more samples (NetStream audio has the same limitation there).

Real-world content: three animations that play AHX and ProTracker modules through sampleData were silent (amplitude 0 over 240 ticks in a local audio_assertions run) and now play, audible on every tick. One of them is the loop from #7879, so this should fix its missing sound; I haven't looked at the slow start reported there. #14815, #14156 and #22912 look like the same gap, but I haven't tested them.

Sound.extract() is still a stub, so content that generates sound from other sounds (e.g. gapless MP3 loops) stays silent; that is a separate PR. Such content now calls the stub from its sampleData listener, though, and the stub allocated a buffer of the requested length, with lengths computed from the sound. That panicked with a capacity overflow on the web build, so the stub now extracts nothing. It still returns undefined: a loop like while (len > 0) len -= sound.extract(bytes, len) then ends, where 0 would make it spin until the script timeout.

Checklist

• I, a human, have self-reviewed this PR and fully understand the changes within.
• I have made or updated tests where possible.
• All of my commits are properly scoped, compile successfully, and pass all tests.
• This PR does not make sense to split up into smaller PRs.
• An LLM was involved in the authoring of this code.

[torokati44]

#14203

[torokati44]

Could you please cherry-pick the 5 commits from the PR linked above that just add tests, and see whether they also pass on this branch?

[torokati44]

Or even vice versa, whether your tests show something wrong in that other PR - but you don't have to check that, it's just my curiosity.

Also, it's possible that of those 5 tests, some test entirely the same thing as one of your tests, and there's no need to introduce redundancy in the final form.
Additionally, it's possible that some of those tests check something that's platform- or system dependent in FP, and if so, those should be removed.

[torokati44]

And num_ticks should always be preferred instead of num_frames (right, @kjarosh?).
I know, some of my old tests also do this wrong, and the test harness drives tests differently because of this, which is not right.

[torokati44]

Purely for inspiration, don't take it over as is, it's a mess: https://github.com/torokati44/ruffle/tree/einsz-avm2-sound-sampledata

[einsz]

Thanks! Your tests found three differences, which I checked in Flash Player 32 (Linux projector). Your branch matches my measurements for load() and the start time. I've pushed my versions, with your _load() idea:

• load() throws 2037 once play() has started generating samples, even if play() returned null because no channel was free.
• The start time is added to SoundChannel.position and to the positions of events after play() returns, but not to playback: play(1000) takes as long to complete as play(0).
• More events before play() returns than either of our branches had: 1 for 2048-sample blocks, 2 for 2049–2800, 3 for 3000–4300, 4 for 4700–6500, 5 for 6800–8192, and more when smaller blocks follow a larger one. It isn't timing-based. I couldn't find Flash's rule; the formula fits all 61 sequences I measured. This is what makes sample_data_event_data_nulled pass.

New test: avm2/sound_sampledata_play (output from Flash). I took your tests that cover something mine don't (too_many_generated_sounds, generated_sound_play_twice, sample_data_event_data_nulled, sound_sampledata_bytearray, and the num_ticks commit). The other three overlap with mine but pass too.

[torokati44]

FYI @einsz in addition to rebasing this PR onto latest master, I've also squashed some of the num_frames -> num_ticks commits into the ones that add the affected tests in the first place, and removed some LLM references from the commit message of another added test...
Also I'm considering splitting out the stubbification of Sound.extract into its own commit, I hope you don't mind.

[torokati44]

(pinging those who have reviewed #14203 in the past)