You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on May 20, 2019. It is now read-only.
Repository navigation
This repository was archived by the owner on May 20, 2019. It is now read-only.
the main idea being that we can take advantage of all the awesomeness that is built on top of npm e.g. https://nodei.co/https://david-dm.org/ many CLI tools etc etc
@sballesteros like @maxogden said :-) Sorry for vagueness here - aim is to reuse npm so we want npm-style registry.
In terms of more general stuff if you have surplus hacking energy I suggest looking at https://github.com/okfn/data.okfn.org which already does some catalog style stuff (won't be registry since we want to reuse npmjs stuff but is a useful "presentation" catalog ...)
@maxogden totally agree.
I am just playing with something at the pre-publish phase to:
send the actual data (and that only) of the data package listed in dataDependencies in another registry (more adapted to store tabular data) and get urls back
replace the data or path of the datapackage resource by these urls
@sballesteros that is very interesting as we do need somewhere to store data / data packages (for those not up to storing themselves). Do take a look at frictionlessdata/frictionlessdata.io#52 - I'd love to get some storage of data packages in data.okfn.org system and would love to collaborate here ...
@rgrp not sure if it is relevant for frictionlessdata/frictionlessdata.io#52 but I have reached a testable stage for my npm inspired registry for data packages. It uses the same technologies as npm / the npm registry (node.js and CouchDB) but diverges so that:
each resource of a data package could be accessible via it's own URL (npm on the other hand stores modules as tarballs).
the dependencies system is more adapted to a data workflow (where data can be big or small so you might not want to download the data package resources).
The registry can be found here and a dpm2 client can be found here.
This README illustrates the differences with npm (although the client can be used as a post install script of npm).
If you think there are rooms for collaboration let me know!
@sballesteros just realized i had a typo - should be #okfn not #okfn0. Also i wonder if it may be better to ping on skype for voice etc (i'm rufuspollock) and monday could be good.
One alternative for this I didn't mention (or think of) before is just using CKAN. This already has a lot of the base infrastructure and supports reasonable sync.
@maxogden@sballesteros here's a somewhat "radical" approach to the registry which is based on what I was currently doing for data.okfn.org. Summed up the approach is: "use github as your catalog"
Current approach for data.okfn.org
data packages are stored on github (with datapackage.json in root)
Activity
sballesteros commentedon Nov 14, 2013
Any preferences for the database
CouchDB ? MongoDB? something else ?
sballesteros commentedon Nov 15, 2013
FYI, I started to work on a CouchDB powered registry here.
max-mapper commentedon Nov 15, 2013
sorry for the vagueness, this issue means setting up an instance of the NPM registry, as opposed to using the existing hosted NPM registry
max-mapper commentedon Nov 15, 2013
the main idea being that we can take advantage of all the awesomeness that is built on top of npm e.g. https://nodei.co/ https://david-dm.org/ many CLI tools etc etc
rufuspollock commentedon Nov 15, 2013
@sballesteros like @maxogden said :-) Sorry for vagueness here - aim is to reuse npm so we want npm-style registry.
In terms of more general stuff if you have surplus hacking energy I suggest looking at https://github.com/okfn/data.okfn.org which already does some catalog style stuff (won't be registry since we want to reuse npmjs stuff but is a useful "presentation" catalog ...)
sballesteros commentedon Nov 15, 2013
@maxogden totally agree.
I am just playing with something at the pre-publish phase to:
dataorpathof the datapackage resource by these urlsrufuspollock commentedon Nov 15, 2013
@sballesteros that is very interesting as we do need somewhere to store data / data packages (for those not up to storing themselves). Do take a look at frictionlessdata/frictionlessdata.io#52 - I'd love to get some storage of data packages in data.okfn.org system and would love to collaborate here ...
max-mapper commentedon Nov 15, 2013
ping @dominictarr who is working on content addressable blob storage in node
sballesteros commentedon Dec 17, 2013
@rgrp not sure if it is relevant for frictionlessdata/frictionlessdata.io#52 but I have reached a testable stage for my npm inspired registry for data packages. It uses the same technologies as npm / the npm registry (node.js and CouchDB) but diverges so that:
The registry can be found here and a dpm2 client can be found here.
This README illustrates the differences with npm (although the client can be used as a post install script of npm).
If you think there are rooms for collaboration let me know!
rufuspollock commentedon Dec 19, 2013
@sballesteros this looks great. I had also come to the view that there would be some divergence with npm and your approach looks excellent.
If you were up for it I'd love to merge dpm2 and dpm into one library and tool (be happy for you to be lead dev here with support from me!)
Are you around at some point to have a quick chat e.g. on irc (#okfn or skype or hangout ... (e.g. how about 5:30pm London today ...)?
sballesteros commentedon Dec 20, 2013
@rgrp great news! quick chat on irc would be perfect. Will hang out on #okfn0.
rufuspollock commentedon Dec 21, 2013
@sballesteros just realized i had a typo - should be #okfn not #okfn0. Also i wonder if it may be better to ping on skype for voice etc (i'm rufuspollock) and monday could be good.
sballesteros commentedon Dec 21, 2013
@rgrp monday on skype it is. I am sebastien.ballesteros. I am in New York for the time zone.
rufuspollock commentedon Dec 28, 2013
One alternative for this I didn't mention (or think of) before is just using CKAN. This already has a lot of the base infrastructure and supports reasonable sync.
rufuspollock commentedon Dec 31, 2013
I think one action from chat yesterday is to think through requirements for the registry e.g.:
rufuspollock commentedon Jan 1, 2014
@maxogden @sballesteros here's a somewhat "radical" approach to the registry which is based on what I was currently doing for data.okfn.org. Summed up the approach is: "use github as your catalog"
Current approach for data.okfn.org
So:
Proposed approach
We could extend this to an approach where we don't have to explicitly list all the data packages and essentially juse Use github for your registry"
rufuspollock commentedon Feb 8, 2014
Fly in the ointment: http://developer.github.com/changes/2013-10-18-new-code-search-requirements/ - you can't do code searches without limiting to a particular user, org or repo
rufuspollock commentedon Feb 14, 2014
@sballesteros would you be happy for me to take a stab at this?
sballesteros commentedon Feb 16, 2014
@rgrp sure, I am still deep into experimenting with a full hypermedia solution.
rufuspollock commentedon May 18, 2014
Now have a list of all data packages on github here https://github.com/datasets/registry/blob/master/github-list.txt
rufuspollock commentedon May 27, 2014
@maxogden further thoughts there - at the moment the approach of using github as the "registry" seems quite workable. wdyt?
pwalsh commentedon May 26, 2016
@rgrp how does this issue relate to the current DP registry? I'm wondering whether to close this, or if there are specific things here for
dpmrufuspollock commentedon May 29, 2016
INVALID. not specific to dpm.