Repository navigation
Registry/Catalog of Community Data Packages #52
Description
We want to allow people to register the data packages they've created in a "community" catalog portion of the site.
Url structure like: /community/{username}/{dp-name}
Options for Implementatoin
Github proxy option (see below)
_Note this option is what is currently (partially) implemented (see comments below for more info)_
- data package at github.com/{username}/{repo} shows up at data.okfn.org/community/{username}/{repo}
-
data.okfn.org/community has some info!(no longer relevant as moved to /data) -
data.okfn.org/community/{username} gives a nice listingnow Nice Owner/User page in data section #111
Full option
- login approach decided (github/twitter/...)
- where do we store info (s3, db, ...)?
- What do you register (path do a datapackage.json ...)
Other related things
Creation of data packages (for you)
For example:
- Provide a link to a published gdoc page and a data package is created for you (where is it stored? github, s3, database?) and webpage is created
Activity
rufuspollock commentedon Jul 28, 2013
Thought: why not just run off github directly at present i.e. just search a given users repos for one's with a datapackage.json in the root?
Idea: you have github.com/xyz/my-data-package => data.okfn.org/??/xyz/my-data-package (re ?? see below)
No need to register anything, no need to do anything!
Aside: what about gist support?
Url structure (separate)
hub is shorter but less clear than community. simplest is first option. I think value of privileging data is sufficient atm.
[#52,community][m]: start on community catalog proxying off github at…
rufuspollock commentedon Aug 2, 2013
We have our first two:
d2m commentedon Aug 2, 2013
@rgrp very cool indeed!
anyway, something is wrong. looking at http://data.okfn.org/community/rgrp/wheat-us
i see right hand a link "Home page:" github.com/datasets/wheat-us
which is clearly wrong
the link under the button "Data Package Info" points to
http://data.okfn.org/community/rgrp/wheat-us/datapackage.json
which reports
"Cannot GET /community/rgrp/wheat-us/datapackage.json"
hth, michael
rufuspollock commentedon Aug 2, 2013
@d2m thanks for the quick spots - these are known bugs and they are getting fixed :-)
[tools,dataset][s]: use homepage in templates and set homepage if not…
[#52,community][s]: main download link now works (on tools/view as we…
rufuspollock commentedon Aug 9, 2013
@d2m both the issues you mention should now be fixed!
rufuspollock commentedon Aug 20, 2013
I think it is becoming clear we also need to allow creation / registration of data packages cf #66. Thoughts:
Still not clear how this fits in the namespace we have if using github proxy through. Perhaps we need to move from that and store stuff somewhere ourselves (e.g. s3 flat file would be possible)
rufuspollock commentedon Sep 7, 2013
Bump to top priority.
rufuspollock commentedon Oct 5, 2013
@davidmiller + @rossjones @psychemedia interested in your thoughts here. Various options as I see it atm:
User story:
rufuspollock commentedon Oct 24, 2013
@mihi-tr I'd be interested in your thoughts here. What workflow would you like?
mihi-tr commentedon Oct 27, 2013
I do like the github route. One problem: the datasets will be indexed on demand only. (as I understand it). There is no clear route to get a new dataset published there. (e.g. a full list of community built data packages).
One way to do this could be to cache datasets that are requested and show them as a list of datasets. (what happens if the datasets 404?)
What speaks against going down the Github route: You'd need a github repo for everything - no possibility to publish datasets easily otherwise...
rufuspollock commentedon Oct 27, 2013
@mihi-tr you've highlighted the 2 major issues:
Neither of these seem show-stoppers and seem a reasonable way to go ...
mihi-tr commentedon Oct 28, 2013
@rgrp One thing I'd love to see in the future is data portals (CKAN) and other sources publish data packages - if we limit ourselves to github we'll include those to be included in a general catalog. But maybe I'm wrong there and this is not the aim for the community repo.
pietercolpaert commentedon Oct 28, 2013
ODI is also doing a lot of work in this direction. Does this sync with theirs? /cc @Floppy
Floppy commentedon Nov 4, 2013
We're doing some stuff, but not in the specific area. Whatever we do next though, we'll be sure to look at what's already out there. Regarding the listing creation thing, a lot of source code ecosystem apps will add a listing for some code when the github repo is entered by the user, so it's a sort of on-demand thing normally.
rufuspollock commentedon Nov 5, 2013
@Floppy would you consider contributing here - contributions would be welcome :-) !
Floppy commentedon Nov 5, 2013
If I get the opportunity, I certainly will :)
[#52,registry][m]: move to having a single common listing for all dat…
[#52,registry][s]: tweak data index page to show community and core d…
[#52,misc][s]: some additional tweaks as part of refactor to have com…
[#52][s]: add in redirects for old data urls /data/{name} to new /dat…
[#52][s]: redirect datapackage.json for old core datasets to new loca…
rufuspollock commentedon Jun 7, 2014
Only remaining thing now is to have /data/{username} work sensibly.
rufuspollock commentedon Jun 7, 2014
FIXED. I'm closing as fixed. the User page is rudimentary and proper one is now #111