Introduce a registration state for faster user responses

[hynky1999]

Problem

Today, user-facing responses appear to wait until enough downstream pipeline or ledger work has completed to report a later state. That makes the system feel slower than necessary, even when the request has already been accepted and recorded for processing.

Proposal

Introduce a new state named registration that represents the point where a user request, job, or pipeline run has been accepted and registered, but has not yet moved into active execution.

Expected behavior:

• Return a response to the user as soon as registration succeeds.
• Persist enough identifying metadata during registration for later tracking, retry, and inspection.
• Transition from registration into the existing execution lifecycle states once workers begin processing.
• Make failure during registration explicit, rather than reporting a later execution failure state.

Why this helps

This separates request acknowledgement from execution progress. Users should see a response much sooner because the API/UI can acknowledge registration without waiting for scheduling, reader setup, processing, or ledger backend work that belongs to later states.

Execution-model context

Comparable systems generally separate acceptance from execution:

• Spark: job submission returns before all stages execute; execution state is tracked separately by the driver/scheduler.
• Beam/Dataflow: pipeline submission and job creation are distinct from worker execution.
• Daft: lazy execution planning keeps registration/planning separate from materialization.
• Hugging Face Datasets: dataset transformations often register a plan/cache operation before expensive materialization.
• Ray/Ray Data: task/object refs acknowledge scheduled work before result availability.

For Refiner, a dedicated registration state is a smaller change than adopting a full scheduler abstraction, while still making the API semantics clearer and improving perceived latency.

Design questions

• Which object owns the registration state: pipeline run, ledger record, worker task, or a shared run state enum?
• What metadata must be durable before returning the early response?
• Should registration be visible in public APIs/docs, or treated as an internal transitional state?
• What are the allowed transitions from registration, especially on validation failure or worker startup failure?

Acceptance criteria

• A new registration state is added to the lifecycle model.
• User-facing calls can return after registration succeeds, before execution begins.
• Existing state transitions are updated and covered by tests.
• Documentation explains the new lifecycle behavior and what users can expect from early responses.