Repository navigation
Replies: 1 comment
|
One idea - what if the service element is generalizable like Environment to Queue Environment (Deadline specific). Can we also have a define the specification to support a Queue Service? |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Background
Many parallel workloads depend on a shared service to provide functionality like coordination or cached data. While you can deploy such a shared service and provide access by deploying infrastructure for it as part of your compute cluster, that puts the endpoint the Tasks depend on outside the tooling parseable job template. It would be useful to specify these services directly in job templates instead. A minimal specification for a service looks very similar to the already-defined step or environment script, so providing a new entity similar to the Environment looks like a promising way to do this.
Some concrete workloads that motivate this:
This idea also relates to an active discussion, #136: Idea to add a [Monitoring] extension, which observes that the runtime currently cannot tell whether a running Action's process is still healthy. A long-lived Service endpoint would benefit from that same health monitoring, so it's another consumer of whatever that idea produces.
The RFC Idea
The idea is to define a new Service entity that is like the
<Environment>entity, but that opens a network port that other entities within its scope, like Environments, Steps, and other Services, can access. Like Environment hasjobEnvironmentsandstepEnvironments, we definejobServicesandstepServices.A Service differs from an Environment in four ways that the schema needs to make first-class:
jobService; the set of Tasks for that Step, for astepService) rather than running to completion up front. The scheduler starts it, keeps it alive while dependent Tasks run, and stops it once the scope completes.Service.*format-string scope, to every dependent entity, so a Task on another host knows where to connect.restartablesetting.Everything else, such as
name,description, embedded files,variables, thelet/expression scoping, and cancelation methods, is intended to mirror<Environment>so the entity is immediately familiar and reuses existing machinery.Lifetime and placement
The defining characteristic of a Service is that it isn't tied to the host running the Tasks that use it, and it outlives any single Session:
jobServiceis scoped to the Job. The scheduler brings it up before scheduling any Task of the Job, keeps it running while the Job's Tasks execute (across however many hosts and Sessions those Tasks occupy), and tears it down once the Job no longer needs it.stepServiceis scoped to a single Step. It is available only to that Step's own Tasks, not to any other Step (including Steps that depend on it), and it is torn down once that Step's Tasks are done.Because the consuming Tasks may run on other hosts, the service's endpoint must be reachable over the network between Worker Hosts, and the scheduler is responsible for keeping the service alive for the duration of the scope. The scheduler is responsible for wiring up the connectivity, by running hosts on the same network or by using port forwarding.
Placement is up to the running system, though. A distributed render manager might dedicate a host to the service, whereas the OpenJD CLI's
openjd runcommand runs an entire job on a single host and would simply start the service alongside the Tasks there.Example: a Valkey store shared across a Job's Tasks
A sketch of how a Service could look
The new pieces
Most of the entity is illustrated in the sketch above. A few more details:
jobServices/stepServicesand<Service>. Ordered lists on the Job Template and Step Template, mirroringjobEnvironments/stepEnvironments; astepServiceis scoped to its one Step. Names must be unique within their scope, and ajobServicename must not collide with astepServicename in the same Job. A<Service>mirrors<Environment>(name,description,variables, embedded files) but addsports,restartable, andhostRequirements, and uses a long-livedrunaction in place ofonEnter/onExit.portsandreadiness. Ports are runtime-allocated by default, or an author may request a specific port. Each declares areadinessprobe the runtime uses to gate dependent Tasks; sketched options areTCP_CONNECT, aCOMMANDprobe (run a command until it exits 0, à la a Dockerdocker pscheck), or anopenjd_service_ready:stdout line consistent with the existingopenjd_*protocol.attr.worker.preemptible. A new standard boolean-valued attribute capability, so a Service (or any Step) can requireattr.worker.preemptible anyOf ["false"]to stay off capacity that could be reclaimed mid-scope, like EC2 Spot Instances. Workers advertise the value; this proposal just reserves the name and meaning.The
Service.*format-string scope. New@fmtstring[host]value references (resolved at execution time, since the endpoint isn't known until the scheduler places the service, and it may change across a restart):Service.<name>.<port>.portandService.<name>.<port>.bindAddress, for the service process, telling it the port and interface to listen on.Service.<name>.<port>.connectAddress, for dependent Environments, scripts, and Tasks, telling them where to connect.All reactions