name and id as identifiers for Data Packages

@amercader

Currently Data Packagese must have a name attribute but do not have an id attribute.

There has been debate about both the semantics (e.g. uniqueness) of the name field and its usability for certain cases (e.g. importing datasets into a new catalog) - see #220 for extensive discussions.

Proposal

Two identifier fields:

name: SHOULD be present (and certainly required for installation etc). Name is human meaningful and is designed to support both resolution (protocol to be determined) and easy use by humans e.g. in data dependencies
- (?) Have this as a MUST?
id: MAY be present. If present MUST be globally unique. Propose it is a 36 bit uuid or similar.

What is the structure of `name`?

name may only contain lower case alphanumeric plus _-. and / as a separator (?? should we allow other url compatible values e.g. :?)

Option 1 - 3 part

Name has the following structure:

[registry/[owner-or-namespace/]]local-name

The primary Data Package registry (assuming there is one) will have the special registry name dp

local-name MUST NOT contain a '/'

# single-part - for resolution one would anticipate these implicitly become
# `{primary-registry}/core/{name}`
finance-vix

#2 part: `registry/local-name`
# Propose that namespace MUST
# either come from a designated central data package registry if / when we have one e.g. `core/gdp`
# OR be a valid domain name e.g. `data.gov.uk/my-name` (so we can piggy back on domain name issuance)
datahub.io/xyz
data.gov.uk/xyz

#3 part:
doi/{doi}   # {doi} usually has /
github.com/rgrp/court-decisions-gb

Asides

I did think about having an initial "scheme" value e.g. dp/core/abc or www/data.gov.uk/xyz but felt we were starting to reinvent the url wheel a bit too much ...*
one option I thought about was about keeping name single-valued and having id support the multipart option.*
What about just using DOI? Ans: DOI requires a relatively complex registration process in order to able to issue DOIs. We want anyone to be able to create data packages
Why not just use URIs / URLs? That is an option and we should think move about it. Main disadvantages are:
- They are somewhat cumbersome
- Are liable to breakage e.g. if a registry simply moves url ... (but that may creates problems with the above too?)
- Do not translate well to local installation
- Implicitly creates relation between name and URL resolution -- what happens if you don't control any url space?

Use Cases

Why does having an identifier matter? What is used for? At the moment the use cases are not very clear.

To refer to in e.g. dependencies dataDependencies field
- could we just use urls? Probably: see http://dataprotocols.org/data-package-identifier/
To use in tooling e.g. dpm install {data-package-name}
- could we just use urls or some other more complex identifier? Probably - see http://dataprotocols.org/data-package-identifier/
Discovery: using an identifier we can locate a data package [in a registry]
- Question: Which registry - or more generally what is the "resolution" protocol. See http://dataprotocols.org/data-package-identifier/ for more on this
To support storing and management in a catalog or registry
- Online e.g. CKAN
- Or locally e.g. .datapackages or similar - a local store or cache

Note also @amercader comment: "As a Catalogue / Registry / Command Line Utility I Want Data Packages to have a global unique id So That I can sanely decide if a Data Package is the same as another one." -- though my question is why do you want to decide if it is the same?

Context

Check out Zooko's Triangle. For names hard to have more than 2 of:
- meaningful (for humans)
- decentralized
- secure / non-colliding

Aims for name:

be human-usable and usable in dependencies
make possible and likely but not guarantee non-collision
be partially distributed

Content-based naming / addressing

One attractive approach to naming that is both secure and decentralized is content-based naming based on hashes. The basic idea is you name content via the (e.g. sha1) hash of the content.

This is attractive and clever but does have 2 drawbacks:

The name changes if the content changes (this could be a feature rather than a bug)
The name is an opaque long string

@danfowler

@danfowler @pwalsh @mfenner @jbenet - any thoughts?

I think making name SHOULD and id MAY means that we've removed any type of identifier from data packages, by default. TBH I'm not sure what is best... but I think if id is not becoming a MUST, then we should leave name a MUST.

@rgrp

@rgrp thanks for summarizing the issues again, and for proposing a path forward. My view is slightly different, summarized as follows:

I strongly recommend to require a globally unique identifier for every data package
the easiest way to do this, in particular without any registration, is to use a UUID
human-readable name identifiers are common for software packages. I think the use case for data is different, and a id is a better fit. One example would be hundreds of datasets only differing in the time window or geolocation they describe (e.g. gold prices). Hard to imagine how to find a common naming scheme for cases such as this one
urls are very good unique identifiers, as they are actionable and have a name space. One important disadvantage is that a lot more infrastructure is required to generate URLs

In short my view is to

MUST have a globally unique id
MUST have a name which does not need to be globally unique (i.e. the current spec)
MAY have one or more 'url` which can point to a registry

@Karissa

OK. Here's the updated proposal:

id: introduce this field and define semantics as globally unique.
- Qu: what is syntax? Do we allow anything or restrict to simple "slug" stuff -- this would exclude e.g. doi or use of URL as id
- This would be SHOULD not MUST as we want to allow people to create ultra-simple data packages without knowing how to create a UUID. However emphasize a MUST if you want to publish to e.g. registries.
name: probably keep this very simple as it is (at most allow a '/" in it - probably not as breaks sluggability ...)
- Qu: make this MUST or SHOULD? as per @Karissa question in 'name' in datapackages shouldn't be required #220

All the fancy multi-part stuff is really a bit of a confusion here as it is really about "package identifiers" for use in e.g. installation of dataDependencies. This could / should go into: http://dataprotocols.org/data-package-identifier/

@mfenner @Karissa @danfowler @pwalsh any thoughts here before we move to implement.

@amercader

Note also @amercader comment: "As a Catalogue / Registry / Command Line Utility I Want Data Packages to have a global unique id So That I can sanely decide if a Data Package is the same as another one." -- though my question is why do you want to decide if it is the same?

Basically so you can decide whether to create a new entry on the registry or update an existing one.

If id needs to be globally unique, then allowing it to be a slug without a centralized registry is not really going to work, so I would strongly suggest that the spec recommends using UUIDs, an URI or another unique identifier (like the Figshare id mentioned). We can add a small note along the lines of "If unsure, use an UUID generated on this site or this one")

@amercader

Encouraging the use of a DOI would be ideal, but any unique id scheme
(e.g., uuid) work work

On Friday, January 29, 2016, Adrià Mercader notifications@github.com
wrote:

Note also @amercader https://github.com/amercader comment: "As a
Catalogue / Registry / Command Line Utility I Want Data Packages to have a
global unique id So That I can sanely decide if a Data Package is the same
as another one." -- though my question is why do you want to decide if it
is the same?

Basically so you can decide whether to create a new entry on the registry
or update an existing one.

If id needs to be globally unique, then allowing it to be a slug without
a centralized registry is not really going to work, so I would strongly
suggest that the spec recommends using UUIDs, an URI or another unique
identifier (like the Figshare id mentioned). We can add a small note along
the lines of "If unsure, use an UUID generated on this site
http://mozilla.pettay.fi/cgi-bin/mozuuid.pl or this one
http://www.guidgen.com/")

—
Reply to this email directly or view it on GitHub
#237 (comment)
.

Karissa McKelvey
http://karissa.github.io/

OK. So current new language looks like:

* `id` - an identifier string for this package. If present, this MUST be
  globally unique and persistently so. For example, it could be a [UUID][], a
  [DOI][]. or even a URI/URL (under a domain you control).

  The id SHOULD be invariant, meaning that it SHOULD NOT change when a data
  package is updated, unless the new package version should be considered a
  distinct package, e.g.  due to significant changes in structure or
  interpretation.

  *Relationship to `name`*: `id` and `name` both serve as identifiers for a
  package. ...

Having written this much I realise there is a bit of a challenge explaining to the average reader of the spec why there are two "identifier" fields and their relationship. Questions so far:

If id is a SHOULD I think name probably should be too.
MUST one of id or name be present?
I am struggling now to explain why we have name and/or its relationship to id. Traditionally e.g. in software packaging name would be your unique, meaningful identifier. We could preserve some of this use here if we had a central registry (which we sort of but do not really have yet). I also feel it is pretty useful for some purposes (e.g. as a file name when installing on disk etc) and as a starting point for many people compared to a UUID.
- One stab at an explanation: "name is human-readable and unique within a given registry (e.g. the registry this data package comes from). However, we anticipate data packages being published to and from multiple registries. As such, name may not be globally unique. id is globally unique but is likely human-opaque."

Yes, I like that explanation.

It would be nice if they were both 'SHOULD'

On Sun, Jan 31, 2016 at 3:58 PM, Rufus Pollock notifications@github.com
wrote:

OK. So current new language looks like:

id - an identifier string for this package. If present, this MUST be
globally unique and persistently so. For example, it could be a [UUID][], a
[DOI][]. or even a URI/URL (under a domain you control).

The id SHOULD be invariant, meaning that it SHOULD NOT change when a data
package is updated, unless the new package version should be considered a
distinct package, e.g. due to significant changes in structure or
interpretation.

Relationship to name: id and name both serve as identifiers for a
package. ...

Having written this much I realise there is a bit of a challenge
explaining to the average reader of the spec why there are two "identifier"
fields and their relationship. Questions so far:

If id is a SHOULD I think name probably should be too.

MUST one of id or name be present?

I am struggling now to explain why we have name and/or its
relationship to id. Traditionally e.g. in software packaging name
would be your unique, meaningful identifier. We could preserve some of this
use here if we had a central registry (which we sort of but do not really
have yet). I also feel it is pretty useful for some purposes (e.g. as a
file name when installing on disk etc) and as a starting point for many
people compared to a UUID.

One stab at an explanation: "name is human-readable and unique
within a given registry (e.g. the registry this data package comes from).
However, we anticipate data packages being published to and from multiple
registries. As such, name may not be globally unique. id is
globally unique but is likely human-opaque."

—
Reply to this email directly or view it on GitHub
#237 (comment)
.

Karissa McKelvey
http://karissa.github.io/

This is complicated.

If name MUST be unique within a given registry (based on last explanation text), this may only be knowable after the creation of a Data Package, and even after possible publication to other registries under the same name.

I understand the logic of all the various arguments here, but I really think we will fall into a hole by trying to please all and having both name and id, where each are a type of identifier.

@pwalsh

I agree with @pwalsh. This is a bit too complicated. As much as I like the human readability of name, and understand that this is a pattern for software packages, I don't see this pattern as a good fit for data packages. While we rarely have more than a few software packages that do the same thing, we can easily have dozens or hundreds of data packages all describing UK gold prices or weather in Paris. If we drop the idea that we have an identifier that is both human readable and globally unique, things become easier, and we can drop name.

The problem with identifiers that first need to be registered, e.g. DOI, or name that is unique in a registry is that this creates overhead when the package is created. For the use cases of data packages that I see, I want to be able to create them without this overhead, e.g. when I have no network connection.

My suggestion is to

require id, and make this a uuid
allow additional identifiers, including DOIs in another field, named for example other_ids. We can also put human readable identifiers such as name in here
drop name, as it is confusing if we also have id
use title for a human readable description, but this doesn't have to be unique

I agree with @rgrp that

The id SHOULD be invariant, meaning that it SHOULD NOT change when a data package 
is updated, unless the new package version should be considered a distinct package, 
e.g. due to significant changes in structure or interpretation.

@pwalsh

I agree with you until you advocate to use title instead of name. Itd be
nice to keep the functionality you propose with title, but keep name as the
keyword. This way, it doesn't break backwards compatibility.

On Sunday, January 31, 2016, Martin Fenner notifications@github.com wrote:

I agree with @pwalsh https://github.com/pwalsh. This is a bit too
complicated. As much as I like the human readability of name, and
understand that this is a pattern for software packages, I don't see this
pattern as a good fit for data packages. While we rarely have more than a
few software packages that do the same thing, we can easily have dozens or
hundreds of data packages all describing UK gold prices or weather in
Paris. If we drop the idea that we have an identifier that is both human
readable and globally unique, things become easier, and we can drop name.

The problem with identifiers that first need to be registered, e.g. DOI,
or name that is unique in a registry is that this creates overhead when
the package is created. For the use cases of data packages that I see, I
want to be able to create them without this overhead, e.g. when I have no
network connection.

My suggestion is to

require id, and make this a uuid

allow additional identifiers, including DOIs in another field, named
for example other_ids. We can also put human readable identifiers such
as name in here

drop name, as it is confusing if we also have id

use title for a human readable description, but this doesn't have to
be unique

I agree with @rgrp https://github.com/rgrp that

The id SHOULD be invariant, meaning that it SHOULD NOT change when a data package is updated, unless the new package version should be considered a distinct package, e.g. due to significant changes in structure or interpretation.

—
Reply to this email directly or view it on GitHub
#237 (comment)
.

Karissa McKelvey
http://karissa.github.io/

Agree about keeping name for backwards compatibility.

@mfenner

@mfenner +1 with the addition of @Karissa re name.

I just don't see any other way forward that will not be confusing.

@rgrp

+1 on good description of id @rgrp
+1 to @mfenner's comments and +1 on using "name"
please always make doi optional, as another attribute.
for id, please use uuid4 or just a large cryptographically random string.
if you use uuid, consider no dashes for copy-paste friendliness -- dashes tend to be confusing too.

@mfenner

+1 @mfenner @Karissa on requiring id (MUST) while also keeping name for backwards compatibility. I suppose this means that name should, going forward, be a SHOULD property?

One comment, if id is required to be a UUID, why not just explicitly make the field name uuid? I also like the other_ids approach, and suggest an object like so:

"uuid": "ff28b27c-6556-45fd-9d17-13368ae6fd44",
"other_ids": {
  "doi": "...",
  "uri": "..."
}

@rgrp

@rgrp @frictionlessdata/specs-working-group

On reflection, I think my last suggestion is ridiculous, as by trying to please every opinion, it serves no one particularly well.

I'm starting to swing towards what I think is the view of @rgrp being that the spec does not need a unique identifier as part of it: it is a platform-specific concern (federated or otherwise).

What we could do is reserve id as a protected property for use as a unique identifier by implementations, and then see how it is used, or, we could require id to be a UUID, which can be validated, and is the most implementation-neutral of all suggestions I've seen so far.

@frictionlessdata/specs-working-group this has been a persistent issue, well beyond this thread. I'd love to hear additional thoughts.

@rgrp

I'm not sure where the proposal for name landed. At the top @rgrp describes a fairly complex process of assigning names that is much more labor-intensive than generating a UUID, which you can do online if you happen to be generating IDs manually.

If, as @pwalsh writes, we're tending towards not having any unique identifiers, then I'm in favor of eliminating name, which is very confusing.

If we do add id, then I'm in favor that it SHOULD be a UUID, but if some of the ecosystem already uses DOI or other identifiers (dat was mentioned above), then by all means let them continue to use those IDs in a 'grandfathered' way.

@rgrp

I'm starting to swing towards what I think is the view of @rgrp being that the spec does not need a unique identifier as part of it: it is a platform-specific concern (federated or otherwise).

This seems like a good compromise. Global uniqueness will be hard to guarantee and only meaningful within the datapackage space. Allowing the id to be a platform specific unique id will make it easier to use datapackages in those platforms. This would allow id's such as:

"id" : "https://doi.org/10.5281/zenodo.166271"

"id" : "dat://f677bd23661a1d5871e40092268d197c73f213f6b8aefebe01709647cfde9528/"

These IDs will be resolvable within the specific platform but also meaningful when viewing the datapackage outside that platform. It is clear what subspace these IDs come from and where they are guaranteed to be unique.

@joehand

@joehand @pwalsh good suggestions. Next steps are now clear:

Separate issue about creating id field - which is up to users to make unique
Separate issue about making name non-required and more flexible (it is basically there for registries)

AGREED: will do as separate PR:

id field which MUST be unique (e.g. uuid, doi etc)
name field is MAY and can be anything you like within reason ...

Adds ID field. Closes #237

Merge pull request #369 from frictionlessdata/feature/id-name

[dp,#237,#418][s]: reintroduce id field on dp as part of #418 rework …

Navigation Menu

Repository navigation

name and id as identifiers for Data Packages #237

Proposal

What is the structure of `name`?

Option 1 - 3 part

Asides

Use Cases

Context

Content-based naming / addressing

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

name and id as identifiers for Data Packages #237

Description

Proposal

What is the structure of name?

Option 1 - 3 part

Asides

Use Cases

Context

Content-based naming / addressing

Activity

rufuspollock commented on Dec 31, 2015

pwalsh commented on Jan 4, 2016

mfenner commented on Jan 6, 2016

rufuspollock commented on Jan 29, 2016

amercader commented on Jan 29, 2016

okdistribute commented on Jan 29, 2016

rufuspollock commented on Jan 31, 2016

okdistribute commented on Jan 31, 2016

pwalsh commented on Feb 1, 2016

mfenner commented on Feb 1, 2016

okdistribute commented on Feb 1, 2016

mfenner commented on Feb 1, 2016

pwalsh commented on Feb 1, 2016

jbenet commented on Feb 1, 2016

danfowler commented on Feb 3, 2016

11 remaining items

pwalsh commented on Nov 10, 2016

jpmckinney commented on Nov 11, 2016

joehand commented on Nov 11, 2016

rufuspollock commented on Nov 17, 2016

rufuspollock commented on Feb 5, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What is the structure of `name`?