multitrait changes

.github/workflows/docs.yml

@@ -11,6 +11,7 @@ jobs:
   Docs:
     permissions:
       contents: read
-    uses: tskit-dev/.github/.github/workflows/docs.yml@v19
+    uses: tskit-dev/.github/.github/workflows/docs.yml@v21
     with:
       install-slim: true
+      install-slim-branch: "multitrait"

.github/workflows/lint.yml

@@ -10,4 +10,4 @@ jobs:
   Lint:
     permissions:
       contents: read
-    uses: tskit-dev/.github/.github/workflows/lint.yml@v19
+    uses: tskit-dev/.github/.github/workflows/lint.yml@v21

.github/workflows/tests.yml

@@ -12,7 +12,7 @@ permissions:
 jobs:
   packaging:
     name: Python packaging
-    uses: tskit-dev/.github/.github/workflows/python-packaging.yml@v19
+    uses: tskit-dev/.github/.github/workflows/python-packaging.yml@v21
 
   test:
     name: Python
@@ -93,6 +93,7 @@ jobs:
           git clone https://github.com/messerlab/SLiM.git
           mkdir -p SLiM/Release
           cd SLiM/windows_compat/gnulib
+          git checkout multitrait # <-- note multitrait branch!!
           touch --date="`date`" aclocal.m4 Makefile.am configure configure.ac config.h.in Makefile.in
           cd ../..
           cd Release
@@ -104,9 +105,27 @@ jobs:
           pip install uv
           uv sync --locked --group test --no-default-groups
 
-      - name: Install SLiM (macOS / Linux)
-        if: matrix.os == 'macos-latest' || matrix.os == 'ubuntu-24.04'
-        run: micromamba install slim -y
+      # UNCOMMENT THIS when the below is commented again
+      # - name: Install SLiM (macOS / Linux)
+      #   if: matrix.os == 'macos-latest' || matrix.os == 'ubuntu-24.04'
+      #   run: micromamba install slim -y
+
+      - name: Install development SLiM
+      # This should be COMMENTED OUT for release versions,
+      # since this builds SLiM from github head.
+      # Also note that this checks out the multitrait branch!!
+        if: (matrix.os == 'macos-latest' || matrix.os == 'ubuntu-24.04') && steps.cache.outputs.cache-hit != 'true'
+        # If we want to re-build slim from a new commit to the slim repo
+        # we may need to bump the cache key above.
+        shell: bash -l {0}
+        run: |
+          git clone https://github.com/messerlab/SLiM.git
+          mkdir -p SLiM/Release
+          cd SLiM/Release
+          git checkout multitrait
+          cmake -DCMAKE_BUILD_TYPE=Release ..
+          make -j 2
+
 
       - name: Run tests
         run: |

CHANGELOG.rst

@@ -2,7 +2,52 @@
 [1.1.2] - 2026-XX-XX
 ********************
 
-In development
+**Breaking changes**:
+
+- The release of SLiM 6.0, changes to metadata (see below) mean that accessing
+  top-level metadata (e.g., `ts.metadata["SLiM"]`) more than a few times in
+  a script will take a long time. Scripts that previously ran quickly may take a
+  prohibitively long. See the documentation for simple changes that fix the problem:
+  https://tskit.dev/pyslim/docs/latest/previous_versions.html
+
+- The SLiM tree sequence file version number has changed to 1.0. Use `pyslim.update`
+  to convert your tree sequence file to this format.
+
+- Metadata for SLiM's mutations are no longer stored along with the tskit mutations,
+  because mutation stacking allows each tskit mutation to be associated with more
+  than one SLiM mutation. Now, metadata for each unique mutation is stored in
+  top-level metadata, under `ts.metadata["SLiM_mutation_list"]`. The recommended
+  way to access this information is by obtaining the SLiM ID-to-metadata dict
+  returned by `pyslim.mutation_metadata(ts)`.
+
+- Previously, `msprime.sim_mutations` with the `msprime.SLiMMutationModel`
+  would record SLiM metadata along with each new mutation. However, msprime
+  does not modify top-level metadata, and so the method `add_mutation_metadata`
+  should be used after adding SLiM mutations.
+
+- This is a SLiM change, but top-level metadata is now encoded using the `json+struct`
+  codec now provided by tskit (so that the mutation metadata is not too large/slow).
+
+- The top-level and individual metadata schemas now depend on the number of traits
+  in the model. The methods `slim_tree_sequence_metadata_schema` and
+  `slim_individual_metadata_schema` can be used to produce correct schema.
+
+**Bug fixes:**
+
+- In some previous versions, converting files produced by a yet-older version of SLiM
+  to the previously-current file version dropped some information from metadata:
+  nucleotide values for mutations, and pedigree parent IDs for individuals. This only
+  may have affected users using `pyslim.convert(ts)` in a previous version of pyslim
+  on a tree sequence `ts` with SLiM file version prior to 0.9.
+
+**New features**:
+
+- SLiM now includes in metadata information about the effects of mutations on
+  quantitative traits, the values of traits for individuals, and the values of
+  various "tags" defined in SLiM.
+
+- `default_slim_metadata` can now take additional arguments to modify the returned
+  values.
 
 ********************
 [1.1.1] - 2026-03-06

docs/metadata.md

@@ -20,7 +20,7 @@ import random
 random.seed(23)
 
 ts = tskit.load("example_sim.trees")
-tables = ts.tables
+tables = ts.dump_tables()
 ```
 
 ```{eval-rst}
@@ -37,7 +37,7 @@ tables = ts.tables
 ## Overview
 
 SLiM puts SLiM-specific information into the *metadata* for the tree sequence,
-as well as for each populations, individuals, nodes and mutations.
+as well as for each population, individual, node and mutation.
 Here is a quick reference to what information is available:
 see the SLiM manual for the more technical writeup.
 A good way to get a generic metadata example is with {func}`.default_slim_metadata`.
@@ -55,9 +55,26 @@ and `ts.metadata["SLiM"]` contains information about the simulation:
 - `spatial_dimensionality`: for instance, `""` or `"x"` or `"xy"` (etcetera)
 - `spatial_periodicity`: whether space wraps around in some directions (same format as dimensionality)
 - `stage`: the *stage* of the life cycle at which the file was written out (either `"first"`, `"early"`, or `"late"`)
+- `name`: the *name* of this species in SLiM
+- `this_chromosome`: contains, for the chromosome in SLiM recorded in this tree sequence
+    * `id`: SLiM's ID 
+    * `index`: the index of this chromosome in the list of chromosomes
+    * `symbol`: the user-assigned symbol
+    * `type`: specifies inheritance type, e.g., `"A"` for autosome
+- `chromosomes`: (optional) a list of all chromosomes in the simulation,
+    for each containing the same information as for `this_chromosome`
+- `traits`: a list of information for each of the traits:
+    * `index`: the index of the trait in SLiM
+    * `name`: the name in SLiM for the trait
+    * `type`: additive, multiplicative, or logistic
+    * `baselineOffset`, `baselineAccumulation`: a value added to all traits, and whether the effect of substitutions
+      accumulate in that value
+    * `directFitnessEffect`: whether the trait has a direct effect on fitness
+    * `individualOffsetMean`, `individualOffsetSD`: parameters governing the individual-level offsets
+      (i.e., "environment" effects)
 
 **Populations:**
-Information about each SLiM-produced population is written to metatadata.
+Information about each SLiM-produced population is written to metadata.
 The format uses JSON and is extensible, so other keys may be present
 and some keys may be missing (for instance, there are no spatial bounds
 in a nonspatial simulation). The metadata may be `None` for populations
@@ -68,7 +85,7 @@ that SLiM did not use. The keys that SLiM uses are:
 - `description`: a string describing the population
 - `selfing_fraction`, `female_cloning_fraction`, `male_cloning_fraction`, and `sex_ratio`: only present when applicable (e.g., in WF simulations)
 - `bounds_x0`, `bounds_x1`, `bounds_y0`, `bounds_y1`, `bounds_z0`, and `bounds_z1`: the spatial bounds, when applicable
-- `migration_records`: A *list* of entries decribing migration between populations in a WF model.
+- `migration_records`: A *list* of entries describing migration between populations in a WF model.
 
 **Individuals:**
 Each individual produced by SLiM contains the following metadata:
@@ -81,27 +98,91 @@ Each individual produced by SLiM contains the following metadata:
 - `subpopulation`: the subpopulation within SLiM the individual was in at the time the file was written out
 - `sex`: the sex of the individual (either {data}`.INDIVIDUAL_TYPE_FEMALE`, {data}`.INDIVIDUAL_TYPE_MALE`, or {data}`.INDIVIDUAL_TYPE_HERMAPHRODITE`)
 - `flags`: additional information; currently only recording whether the individual was a "migrant" or not (see the SLiM manual)
+- `tag`, `tagF`: the corresponding properties in SLiM: default values returned by pyslim
+  are the special values that SLiM uses to mean that the values are unset
+- `tagL0`, `tagL0_set`, etcetera: again, the corresponding properties in SLiM;
+  the purpose of `tagLX_set` is to record whether the tag has been set in the simulation
+- `per_trait`: a list of information about the trait values for this individual; these are in the same order
+  as the traits listed in top-level metadata;
+    * `phenotype`: the trait value
+    * `offset`: the individual's offset (i.e., the "environmental effect")
 
 **Nodes:**
 Each "node" produced by SLiM (i.e., "genome" within SLiM) has:
 
-- 'slim_id': the unique ID associated with the genome by SLiM
-- 'is_null': whether the genome is a "null" genome (in which case it isn't
+- `slim_id`: the unique ID associated with the genome by SLiM
+- `is_vacant`: records the genome is a "vacant" genome (in which case it isn't
   really there, so shouldn't have any mutations or relationships in the tree
-  sequence!)
-- 'genome_type': the 'type' of this genome (0 for autosome, 1 for X, 2 for Y)
+  sequence!) - see [](sec_overview_vacant_nodes) for more explanation
 
 **Mutations:**
-Each mutation's metadata is a dictionary with a single key, `"mutation_list"`,
-whose entry is a *list* of metadata dictionaries corresponding to the mutations that are "stacked",
-i.e., all present, in all genomes inheriting from this (tskit) mutation.
-So, `ts.mutation(12).metadata["mutation_list"]` is a list, each of whose entries contains:
+Prior to SLiM 6.0, mutation metadata was associated with the tskit mutation objects.
+Now, this is stored in top-level metadata, under ``ts.metadata["SLiM_mutation_list"]``.
+Each entry
 
+- `mutation_id`: the numeric ID of mutation in SLiM
 - `mutation_type`: the numeric ID of the `MutationType` within SLiM
-- `selection_coeff`: the selection coefficient
 - `subpopulation`: the numeric ID of the subpopulation the mutation occurred in
 - `slim_time`: the value of `community.tick` when the mutation occurred
 - `nucleotide`: either `-1` if there is no associated nucleotide, or the numeric code for the nucleotide (see {data}`.NUCLEOTIDES`)
+- `per_trait`: a list of information in the same order as the traits in top-level metadata, recording for each:
+    * `effect_size`: the effect on the trait of this mutation
+    * `dominance`: its dominance coefficient
+    * `hemizygous_dominance`: its hemizygous dominance coefficient (see the SLiM manual)
+- `padding`: this is simply empty bytes, here for byte-alignment reasons, and is always `None`
+
+
+(sec_metadata_using_top_level)=
+
+## Using top-level metadata
+
+If you are going to be using information from top-level metadata,
+it is good practice to extract the metadata as a separate python object once
+and refer to that object, since otherwise you can incur runtime penalties
+for decoding and copying the metadata every time you call `ts.metadata`.
+This can be substantial, given the amount of mutation information
+in top-level metadata.
+For instance, to subtract baseline offsets from individual's trait values,
+we might do:
+```{code-cell}
+md = ts.metadata
+traits = md["SLiM"]["traits"]
+values = [
+    [x['phenotype'] - y["baselineOffset"] for x, y in zip(ind.metadata['per_trait'], traits)]
+    for ind in ts.individuals()
+]
+```
+If we instead inserted ``ts.metadata["SLiM"]["traits"]`` directly into the loop,
+this would become infeasibly slow.
+
+In some more detail:
+each time python evaluates ``ts.metadata`` (e.g., using ``ts.metadata["SLiM"]``)
+a new copy of the metadata dict is decoded and returned. Furthermore, a number
+of `pyslim` functions need to look up information from metadata under the hood.
+For instance, previously it was acceptable to run
+``[pyslim.slim_time(ts, mut.time) for mut in ts.mutations()]``.
+However, this could now easily take hours even for moderately-sized simulations.
+There are several recommendations for how to mitigate this:
+
+- If you use information from top-level metadata, make a copy of it
+  and refer to that copy instead: so, ``ts_metadata = ts.metadata``
+  after ``ts = tskit.load(...)`` and then use ``ts_metadata``. However,
+  be careful that you use the correct metadata object corresponding
+  to the correct tree sequence object!
+
+- Use a single pyslim function call rather than many calls to the same one
+  within a loop. For instance, run:
+  ``slim_times = pyslim.slim_time(ts, ts.mutations_time)`` and extract
+  slim times from this vector. Similarly, use {func}`.nodes_vacant`
+  instead of {func}`.node_is_vacant`.
+
+- Some pyslim methods will accept a pre-extracted metadata dictionary
+  as an optional argument. If this is not provided, those methods will
+  extract the metadata again. The methods that now take a `ts_metadata` argument are:
+  {func}`.individual_ages`,
+  {func}`.individual_ages_at`,
+  {func}`.individuals_alive_at`, and
+  {func}`.slim_time`.
 
 
 (sec_metadata_tools)=
@@ -110,10 +191,20 @@ So, `ts.mutation(12).metadata["mutation_list"]` is a list, each of whose entries
 
 The dictionaries describing the schema for these metadata entries
 are available in `pyslim.slim_metadata_schemas`.
-Furthermore, this method may be useful in working with metadata:
+Furthermore, these methods may be useful in working with metadata:
 
 ```{eval-rst}
 .. autofunction:: default_slim_metadata
+
+.. autofunction:: slim_tree_sequence_metadata_schema
+
+.. autofunction:: slim_individual_metadata_schema
+
+.. autofunction:: slim_node_metadata_schema
+
+.. autofunction:: set_tree_sequence_metadata
+
+.. autofunction:: set_metadata_schemas
 ```
 
 
@@ -125,13 +216,13 @@ see {ref}`tskit's metadata documentation <tskit:sec_metadata>`.
 ### Top-level metadata
 
 The entries of the top-level metadata dict are *read-only*.
-So, you might think that
+So, although you might think that
 `tables.metadata["SLiM"]["model_type"] = "nonWF"`
 would switch the model type,
-but this in fact (silently) does nothing. To modify the top-level metadata,
-we must (a) work with tables (as tree sequences are immutable, and (b)
+this in fact (silently) does nothing. To modify the top-level metadata,
+we must (a) work with tables (as tree sequences are immutable), and (b)
 extract the metadata dict, modify the dict, and copy it back in.
-Instead, you should do
+So, the code for modifying top-level metadata is:
 ```{code-cell}
 md = tables.metadata
 md["SLiM"]["model_type"] = "nonWF"