feat: declare unordered lists whose members are unique

docs/embedding.md

@@ -322,6 +322,9 @@ whatever the unknown values turn out to be:
   Rules that apply in every state are still checked.
 - An unknown list, such as `users`, is an unknown field, and none of its items are checked. An
   unknown item, such as `users[1]`, is skipped while the other items are checked.
+- A list marked `unique` is not checked for repeats while it is unknown, and an unknown member,
+  such as `phases[1]`, is not compared with the others. Two known members that repeat each other
+  are still reported.
 - An unknown `property` skips the check that a property task's property is not one another task
   manages. A path that names no field is ignored.
 

docs/schemas/task-catalog-v1.schema.json

@@ -268,7 +268,7 @@
       "additionalProperties": false,
       "required": ["name", "type", "required"],
       "if": { "properties": { "type": { "const": "list" } } },
-      "else": { "not": { "required": ["ordered"] } },
+      "else": { "allOf": [{ "not": { "required": ["ordered"] } }, { "not": { "required": ["unique"] } }] },
       "properties": {
         "name": { "type": "string", "description": "The recipe key." },
         "type": {
@@ -325,7 +325,11 @@
         },
         "ordered": {
           "type": "boolean",
-          "description": "Whether the order of a list's elements is significant. `true` means docket applies and compares them in order, so a reorder is drift; `false` means docket ignores order. It says nothing about uniqueness. Present on every `list` field and never on any other type; a list without it comes from a docket that predates the key."
+          "description": "Whether the order of a list's elements is significant. `true` means docket applies and compares them in order, so a reorder is drift; `false` means docket ignores order. It says nothing about uniqueness, which `unique` and an identity collection's item identity declare. Present on every `list` field and never on any other type; a list without it comes from a docket that predates the key."
+        },
+        "unique": {
+          "const": true,
+          "description": "The list's members must be distinct: validation rejects a member equal to an earlier one. Present only on a `list`, and never on an identity collection, whose item identity already makes its members distinct, so every list with `ordered: false` is one or the other and can be modelled as a set. Absent on a list that may repeat a member."
         },
         "identity": {
           "enum": ["key", "collection"],

docs/task-catalog.md

@@ -144,6 +144,7 @@ the reference page lists them in.
 | `write_only` | Present and `true` when the server stores the value but a read of the resource never returns it. |
 | `apply_only` | Present and `true` when the value steers how apply runs and is not server state. |
 | `ordered` | On every `list`: `true` when the element order is significant, `false` when docket ignores it. Absent on any other type. |
+| `unique` | Present and `true` on a `list` whose members must be distinct. Absent on an identity collection and on any other type. |
 | `identity` | `key` or `collection`. Absent on a field that is neither. |
 | `item` | The element shape of a `list` or `dict`. Absent on a scalar. |
 
@@ -196,9 +197,17 @@ applies the elements in order and compares them in order, so a reorder is drift:
 positionally onto the service's container ports, and `dokku_service_create`'s network lists are
 compared as written. `false` means docket ignores the order, as it does for `dokku_domains`'s
 `domains` or `dokku_http_auth_allowed_ip`'s `allowed_ips`, and an exported task may return the
-elements in any order. `false` says nothing about uniqueness: `dokku_scheduler_k3s_profile`'s
-`kubelet_args` ignores order but still counts a repeated argument. Every `list` carries the key,
-element fields included, so a list without it comes from a docket that predates it.
+elements in any order. Every `list` carries the key, element fields included, so a list without it
+comes from a docket that predates it.
+
+`ordered` says nothing about uniqueness; `unique` and `identity` do. An identity collection's
+members are distinct by their item identity. Any other list whose members must be distinct carries
+`unique: true`, and docket refuses a recipe that repeats a member, as `dokku_config`'s `preserve`,
+`dokku_storage_mount`'s `phases` (on the task and on each `mounts` item) and
+`dokku_scheduler_k3s_profile`'s `kubelet_args` do. Two members are the same when they are equal,
+so in `kubelet_args` a repeat is the exact same `key=value`; one key with two values is two
+arguments. Every list with `ordered: false` is either an identity collection or `unique`, so a
+consumer can model each of them as a set.
 
 ### Item shapes
 
@@ -239,6 +248,8 @@ exactly what `plan`, `apply` and `docket validate` would.
 
 `required_when` and `forbidden_when` on a field list the states in which it must, or must not, be
 set. The states are values of the task's `state` field, so they apply only to a task that has one.
+`unique` on a list holds in every state: a member equal to an earlier one is reported at the later
+member's path, such as `phases[1]`.
 
 `constraints` on a task, and `item.constraints` on an object item, are the rules over several
 fields, in declaration order:
@@ -442,8 +453,8 @@ markdown are relative to `docs/tasks/`. A reference page prints the Parameters t
 
 `note` on a field is `suffix`, the text the Parameters table appends to `description` as is,
 leading space or period included, so `description` followed by `suffix` is the table cell. It holds
-the item shape of a structured list, `ordered`, `required_when`, `forbidden_when`, `runner_file`,
-`write_only`, `apply_only` and `sensitive`, in that order. Unlike the task notes, neither
+the item shape of a structured list, `ordered`, `unique`, `required_when`, `forbidden_when`,
+`runner_file`, `write_only`, `apply_only` and `sensitive`, in that order. Unlike the task notes, neither
 `description` nor `suffix` is escaped for markdown.
 
 ```json

docs/tasks/dokku_config.md

@@ -33,7 +33,7 @@ Tasks applied at once on the same object take turns; see [Running tasks concurre
 | `app` | string | yes |  |  | Name of the app |
 | `restart` | bool | no | true |  | Flag indicating if the app should be restarted (applied only, not read back) |
 | `config` | dict | no |  |  | Map of configuration key-value pairs (must not be set when state is clear) |
-| `preserve` | list | no |  |  | Keys that state 'set' and 'clear' leave untouched, on top of the service-link keys, NO_VHOST, and the git rev-env-var key they always keep (must not be set when state is present or absent) (applied only, not read back) |
+| `preserve` | list | no |  |  | Keys that state 'set' and 'clear' leave untouched, on top of the service-link keys, NO_VHOST, and the git rev-env-var key they always keep (no duplicates) (must not be set when state is present or absent) (applied only, not read back) |
 | `state` | string | no | present | present, absent, set, clear | Desired state of the configuration: 'present' sets the named keys and 'absent' unsets them, leaving every other key alone; 'set' declares the app's entire config, unsetting every key it does not name; 'clear' unsets every key. Both 'set' and 'clear' keep keys a service link wrote, NO_VHOST, the git rev-env-var key, and any key listed in 'preserve' |
 
 ## Examples

docs/tasks/dokku_scheduler_k3s_profile.md

@@ -26,7 +26,7 @@ Keyed by `name`.
 | --- | --- | --- | --- | --- | --- |
 | `name` | string | yes |  |  | Name of the node profile. Creating one requires a lowercase name of at most 26 characters: dokku prepends dokku-node-sysctls-profile- to it to name the profile's node-sysctls helm release, and helm caps a release name at 53 characters. |
 | `role` | string | yes |  | server, worker | Role for nodes joined with this profile. |
-| `kubelet_args` | list | no |  |  | List of key=value kubelet arguments to forward to k3s. |
+| `kubelet_args` | list | no |  |  | List of key=value kubelet arguments to forward to k3s. (no duplicates) |
 | `taint_scheduling` | bool | no |  |  | Whether to taint the node so only workloads that tolerate the taint schedule on it. |
 | `allow_unknown_hosts` | bool | no |  |  | Whether to allow ssh connections to nodes whose host key is not yet trusted. |
 | `state` | string | no | present | present, absent | Desired state of the profile. |

docs/tasks/dokku_storage_mount.md

@@ -34,7 +34,7 @@ Tasks applied at once on the same object take turns; see [Running tasks concurre
 | `entry_name` | string | no |  |  | Named storage registry entry to attach (must not be set when state is set or clear) |
 | `host_dir` | string | no |  |  | Host directory to mount in the legacy bind-mount form (must not be set when state is set or clear) |
 | `container_dir` | string | no |  |  | Container directory to mount (must not be set when state is set or clear) |
-| `phases` | list | no |  |  | Deployment phases the attachment applies to (deploy, run). Empty defers to the dokku default. (must not be set when state is set or clear) |
+| `phases` | list | no |  |  | Deployment phases the attachment applies to (deploy, run). Empty defers to the dokku default. (no duplicates) (must not be set when state is set or clear) |
 | `process_type` | string | no |  |  | Process type the attachment applies to. With mounts, the process type whose mounts the list describes; mounts of other process types are left alone. Empty means dokku's _default_ scope. |
 | `subpath` | string | no |  |  | Subpath within the entry to mount (must not be set when state is set or clear) |
 | `readonly` | bool | no |  |  | Mount the attachment as read-only (must not be set when state is set or clear) |

docs/writing-tasks.md

@@ -95,7 +95,7 @@ A few conventions to follow:
   everything else is specific to the task.
 - Give every field a `description:"..."` tag. The docs generator reads it (along with `required`,
   `required_when`, `forbidden_when`, `default`, `options`, `sensitive`, `runner_file`, `write_only`,
-  `apply_only`, and `ordered`) to build the task's Parameters table, so a field without one renders an empty description cell. Those same tags are what
+  `apply_only`, `ordered`, and `unique`) to build the task's Parameters table, so a field without one renders an empty description cell. Those same tags are what
   [`docket schema`](task-catalog.md) publishes, so a missing description is a hole in the
   machine-readable catalog as well as in the docs. Add `,omitempty` to the `yaml` tag of optional
   fields so example YAML stays clean, and use `required:"false"` whenever a field has a `default`
@@ -201,6 +201,13 @@ A few conventions to follow:
   [`docket schema`](task-catalog.md#fields) publishes the answer so a consumer can choose a list or
   a set type, the generator marks an ordered field on the task's page, and a list field without
   the tag fails the build.
+- Tag a slice field `unique:"true"` when its members must be distinct and it is not an
+  `identity:"collection"`, whose item identity already makes them so (`dokku_config`'s `preserve`,
+  `dokku_storage_mount`'s `phases`). The constraint checker then rejects a member equal to an
+  earlier one in every state, the catalog publishes `unique: true`, and the task's page notes "(no
+  duplicates)". Every `ordered:"false"` list must be an identity collection or `unique:"true"`, so
+  a consumer can model it as a set; `TestCatalogUnorderedListsAreSets` fails the build otherwise.
+  A list that may legitimately repeat a member has to be `ordered:"true"`.
 - When a task type is deprecated (typically because the underlying dokku subcommand was deprecated
   or a richer replacement task exists), implement the optional `Deprecation() string` method.
   The generator renders the returned message in a Deprecated admonition on the task's page and

internal/tasks/catalog.go

@@ -345,12 +345,20 @@ type FieldSchema struct {
 	// Ordered is present on every list field and absent on every other kind.
 	// True means docket applies and compares the elements in order, so a
 	// reorder is drift (buildpacks, exposed service ports). False means docket
-	// ignores element order. It says nothing about uniqueness: kubelet_args
-	// ignores order but still counts a repeated argument. A consumer choosing
+	// ignores element order. It says nothing about uniqueness, which Unique
+	// and an identity collection's item identity declare. A consumer choosing
 	// between a list and a set type reads it instead of keeping its own table,
 	// and a list without it comes from a docket that predates the key (#605).
 	Ordered *bool `json:"ordered,omitempty"`
 
+	// Unique marks a list whose members must be distinct, from its
+	// `unique:"true"` tag: validation rejects a member equal to an earlier
+	// one. Absent on any other list and on every other kind. An identity
+	// collection does not carry it, since its item identity already makes
+	// members distinct, so every unordered list is one or the other and can
+	// be modelled as a set (#648).
+	Unique bool `json:"unique,omitempty"`
+
 	// Identity is "key" or "collection" when the field carries an identity
 	// tag, and absent otherwise.
 	Identity string `json:"identity,omitempty"`
@@ -724,6 +732,7 @@ func buildFields(rt reflect.Type) ([]FieldSchema, error) {
 		}
 		if schema.Type == TypeList {
 			schema.Ordered = orderedTag(field.Tag)
+			schema.Unique = field.Tag.Get(uniqueTag) == "true"
 		}
 		switch field.Tag.Get(identityTag) {
 		case identityTagKey:

internal/tasks/catalog_notes.go

@@ -123,6 +123,7 @@ type fieldNoteKind struct {
 var fieldNoteKinds = []fieldNoteKind{
 	{true, itemFieldsNote},
 	{false, orderedNote},
+	{false, uniqueNote},
 	{false, requiredWhenNote},
 	{false, forbiddenWhenNote},
 	{false, runnerFileNote},
@@ -281,6 +282,13 @@ func orderedNote(field FieldSchema, _ State) string {
 	return "(order is significant)"
 }
 
+func uniqueNote(field FieldSchema, _ State) string {
+	if !field.Unique {
+		return ""
+	}
+	return "(no duplicates)"
+}
+
 func requiredWhenNote(field FieldSchema, state State) string {
 	if len(field.RequiredWhen) == 0 {
 		return ""

internal/tasks/catalog_test.go

@@ -498,7 +498,7 @@ func TestCatalogOrderedMatchesTags(t *testing.T) {
 
 	// The lists where a reorder is drift, and a spread of the ones where it is
 	// not: plain collections, keyed structured entries, an element struct's
-	// own list, and kubelet_args, which ignores order but counts duplicates.
+	// own list, and kubelet_args, which ignores order and refuses duplicates.
 	for _, tc := range []struct {
 		typeKey string
 		path    []string

internal/tasks/catalog_unique_test.go

@@ -0,0 +1,108 @@
+package tasks
+
+import (
+	"reflect"
+	"slices"
+	"strings"
+	"testing"
+)
+
+// walkCatalogFields walks every task's fields, element fields included, and
+// calls visit with each field's `type.path` name and its struct field.
+func walkCatalogFields(t *testing.T, visit func(at string, structField reflect.StructField, field FieldSchema)) {
+	t.Helper()
+	var walk func(where string, rt reflect.Type, fields []FieldSchema)
+	walk = func(where string, rt reflect.Type, fields []FieldSchema) {
+		for _, field := range fields {
+			at := where + "." + field.Name
+			structField, ok := structFieldByYAMLName(rt, field.Name)
+			if !ok {
+				t.Errorf("%s has no struct field", at)
+				continue
+			}
+			visit(at, structField, field)
+			if field.Item != nil && field.Item.Type == TypeObject {
+				walk(at+"[]", structElem(sliceOrMapElem(structField.Type)), field.Item.Fields)
+			}
+		}
+	}
+	for _, schema := range wholeCatalog(t).Tasks {
+		walk(schema.Type, taskStructType(lookupTask(schema.Type)), schema.Fields)
+	}
+}
+
+// TestCatalogUniqueMatchesTags holds the `unique:` tag to the catalog: it is
+// absent or exactly "true", only on a list that is not an identity
+// collection, and the catalog publishes unique on exactly the tagged fields
+// (#648).
+func TestCatalogUniqueMatchesTags(t *testing.T) {
+	var unique []string
+	walkCatalogFields(t, func(at string, structField reflect.StructField, field FieldSchema) {
+		tag, tagged := structField.Tag.Lookup(uniqueTag)
+		if tagged && tag != "true" {
+			t.Errorf("%s is tagged unique:%q; the tag is unique:\"true\" or absent", at, tag)
+		}
+		if tagged && field.Type != TypeList {
+			t.Errorf("%s is a %s and must not carry a unique tag", at, field.Type)
+		}
+		if tagged && field.Identity == IdentityRoleCollection {
+			t.Errorf("%s is an identity collection, whose item identity already makes members distinct, and must not carry a unique tag", at)
+		}
+		if field.Unique != (tagged && tag == "true" && field.Type == TypeList) {
+			t.Errorf("%s publishes unique = %v; tag says %q", at, field.Unique, tag)
+		}
+		if field.Unique {
+			unique = append(unique, at)
+		}
+	})
+
+	// The lists #648 declared. Adding one is a deliberate edit here.
+	want := []string{
+		"dokku_config.preserve",
+		"dokku_scheduler_k3s_profile.kubelet_args",
+		"dokku_storage_mount.mounts[].phases",
+		"dokku_storage_mount.phases",
+	}
+	slices.Sort(unique)
+	if !slices.Equal(unique, want) {
+		t.Errorf("unique lists = %q; want %q", unique, want)
+	}
+}
+
+// TestCatalogUnorderedListsAreSets holds every list whose order docket
+// ignores to set semantics: it is an identity collection, whose item identity
+// makes its members distinct, or it is unique, so a consumer can model every
+// unordered list as a set (#648).
+func TestCatalogUnorderedListsAreSets(t *testing.T) {
+	unordered := 0
+	walkCatalogFields(t, func(at string, _ reflect.StructField, field FieldSchema) {
+		if field.Type != TypeList || field.Ordered == nil || *field.Ordered {
+			return
+		}
+		unordered++
+		collection := field.Identity == IdentityRoleCollection
+		if collection == field.Unique {
+			t.Errorf("%s is unordered and must be exactly one of an identity collection or unique:\"true\" (collection %v, unique %v)", at, collection, field.Unique)
+		}
+	})
+	if unordered == 0 {
+		t.Fatal("no unordered lists found")
+	}
+}
+
+// TestCatalogUniqueNote checks the "(no duplicates)" field note on each
+// unique list, element fields included, and its absence elsewhere.
+func TestCatalogUniqueNote(t *testing.T) {
+	walkCatalogFields(t, func(at string, _ reflect.StructField, field FieldSchema) {
+		has := field.Note != nil && strings.Contains(field.Note.Suffix, "(no duplicates)")
+		if has != field.Unique {
+			t.Errorf("%s note %v; unique %v", at, field.Note, field.Unique)
+		}
+	})
+
+	catalog := wholeCatalog(t)
+	kubeletArgs := fieldFor(t, schemaFor(t, catalog, "dokku_scheduler_k3s_profile").Fields, "kubelet_args")
+	if kubeletArgs.Note == nil || kubeletArgs.Note.Suffix != " (no duplicates)" {
+		t.Errorf("kubelet_args note = %+v; want suffix %q", kubeletArgs.Note, " (no duplicates)")
+	}
+}

internal/tasks/config_task.go

@@ -27,7 +27,7 @@ type ConfigTask struct {
 	Config map[string]string `forbidden_when:"clear" identity:"collection" yaml:"config,omitempty" description:"Map of configuration key-value pairs"`
 
 	// Preserve lists keys that state 'set' and 'clear' leave in place
-	Preserve []string `required:"false" forbidden_when:"present,absent" apply_only:"true" ordered:"false" yaml:"preserve,omitempty" description:"Keys that state 'set' and 'clear' leave untouched, on top of the service-link keys, NO_VHOST, and the git rev-env-var key they always keep"`
+	Preserve []string `required:"false" forbidden_when:"present,absent" apply_only:"true" ordered:"false" unique:"true" yaml:"preserve,omitempty" description:"Keys that state 'set' and 'clear' leave untouched, on top of the service-link keys, NO_VHOST, and the git rev-env-var key they always keep"`
 
 	// State is the desired state of the configuration
 	State State `required:"false" yaml:"state,omitempty" default:"present" options:"present,absent,set,clear" description:"Desired state of the configuration: 'present' sets the named keys and 'absent' unsets them, leaving every other key alone; 'set' declares the app's entire config, unsetting every key it does not name; 'clear' unsets every key. Both 'set' and 'clear' keep keys a service link wrote, NO_VHOST, the git rev-env-var key, and any key listed in 'preserve'"`