Dataset Versions and Deprecation

Connectomic datasets are not static: reconstructions are re-released as proofreading improves, neurons are split, merged or re-identified, and the resources that host them may occasionally move or shut down. When VFB ingests a new release, it must decide what happens to the entities from the previous version. This page documents those policies so that users understand why a neuron, link or connection may change, disappear, or persist between releases.

Affected entities

A connectomic dataset is represented in VFB by several types of graph node and edge:

Core principle: Site and Neuron deprecation are independent

Whether a Site is deprecated and whether a Neuron is deprecated are decided separately. A Neuron can be valid while its data source is deprecated, and a data source can remain live while individual Neurons within it are retired. The two states are tracked independently and have different consequences (below).

Deprecation does not delete a node. The node and its identifier are retained (so old IDs resolve), but it is marked deprecated and treated accordingly.

What happens when a new dataset version is released

For example, when BANC v626 (the old version) is replaced by v888 (the new version):

• New nodes are created for the new Site and DataSet, and for any new Neurons in the release (accessions that were not previously in VFB). The markers that identify a Site as the live, canonical connectome source — its symbol (the short dataset code listed in EM Data), Connectome label and is_data_source flag — are transferred from the old Site to the new one (removed from the old, added to the new), so the new Site takes over as the canonical source.
• A term_replaced_by edge is added to link the old and new DataSet/Site.
• Old DataSet — deprecated (superseded by the new DataSet, which it links to via term_replaced_by).
• Old Sites that still exist, i.e. links will still resolve — not deprecated.
• Old Sites that no longer exist, i.e. links will not resolve — deprecated.
• Neurons with accessions that persist in the new data — not deprecated. They keep their database_cross_reference edge to the old Site and gain an edge to the new Site.
• Neurons with accessions that are not present in the new data — deprecated. They have no database_cross_reference edge to the new Site; their only cross-reference is to the old, deprecated Site.
• Connectivity edges are replaced with edges from the new data for Neurons that are not deprecated.
• Old images are removed. Images from the new data are loaded for Neurons that are not deprecated.
• Cell type / FBbt links are replaced with annotations based on the new data for Neurons that are not deprecated. These may be removed from deprecated Neurons if they are incorrect based on the new data, or retained if they are still valid.
• Channel nodes are always retained (not deprecated). Deprecation is not necessary, as all queries are keyed off of the Neuron node.

What happens when a dataset/site is retired with no replacement

The Site is deprecated and there is no new Site.

• Neurons — remain valid (not deprecated) and remain valid query targets. Their only data source is now a deprecated Site, so no link can be built to a live resource, but the Neurons, their connectivity and their images are still served.
• The Site — deprecated, but its is_data_source flag remains [true] so the Site stays the canonical source for its Neurons. That attribution is what keeps those Neurons discoverable: they continue to be picked up by queries even though their Site is deprecated.
• The DataSet — not deprecated: with no replacement version to supersede it, it remains the current representation of the data.

Effects on the website and queries

These follow from the states above and are enforced when results are generated:

• Deprecated Neurons are excluded from connectivity results and from the Neuron counts used in connectivity summaries (so they do not appear as partners and do not affect percentages).
• Deprecated Sites never produce a clickable external link. Where a results table has source / accession columns (e.g. instance and similar-neuron tables, connectivity tables), the source name and accession are still shown as plain text — they are just not linked. In the Term Info cross-reference list, whose entries exist specifically to be links, a deprecated Site’s entry is omitted.
• Neurons whose only data source is a deprecated Site remain valid query targets and are returned normally; only their outbound link to the dead resource is suppressed.

Technical note

Both is_data_source and the deprecation flag are stored as list-valued annotations in the database (e.g. is_data_source = [true]). Deprecation also surfaces in the search index as Deprecated within the node’s types list. Result-generating code keys off these annotations to apply the behaviours above; Site and Neuron deprecation are checked independently.

See also

• EM Data — the integrated connectomic datasets and their versions.
• Resources — the external sites/resources that host the data.
• Term Info — where cross-references and connectivity are shown.