You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The more I find ideas to improve the sizes of geobuf, the more I realize that we would need to completely rewrite the schema to support the improvements, breaking compatibility. So I'm opening the ticket to start a discussion about what a perfect Geobuf schema would look like (not to say this is a priority, but still a good thing to discuss).
The main difference is utilizing the power and flexibility of the new oneof statement to create a better and more compact schema.
Features:
the data itself contains information whether it's Feature, FeatureCollection, Geometry or GeometryCollection, so you don't need to guess this when decoding
keys and values for properties are stored separately in the top-level Data object, and features only store indexes to them (like vector-tile-spec does); keys and possibly values are reduced to unique values Fancy encoding for properties #26 — for much better properties packing
geometry coordinates are a oneof set of different fields (depending on type), which solves the ambiguity with null values since a oneof field can be empty (without a default value), and also makes it easier to understand and work with
feature has a oneof of either geometry or geometry collection instead of repeated geometries
coordinates are stored as delta-encoded sint32 to take full advantage of varint and zig-zag encoding Switch to sint32 for coords #24 — for much more compact geometries
Value message is also a oneof of different value types
the property value int type is sint32 instead of int64 — it's more compact and JS doesn't actually handle int64 values; in addition, uint type becomes uint32
the data contains optional flag indicating whether it contains altitude (third coordinate), since this is a global-level setting
What about letting the feature id field leverage oneof to allow encoding as uint32 values (much more common than string IDs in many cases we deal with):
@brendan-ward yes, excellent suggestion! Added it to the proto gist, except including oneof directly in the Feature message, since oneof fields are already optional by nature.
@tmcw sure, I just wanted to share the motivation before suggesting that I completely rewrite this repo from scratch while breaking compatibility, which can be very annoying. :)
I don't mind anything being rewritten ever as long as it's better :) But the people who might feel the pain of this would be @mick basically, because it's used in cardboard afaik
@kkaefer depends on the data — Geobuf is a GeoJSON equivalent, without encoding topology. It will probably be smaller compared to TopoJSON, but I'm thinking about making it support TopoJSON encoding as well for even more compression (or making a separate protobuf schema for TopoJSON).
I would just chime in to say that I imagine many implementations of geobuf will use it as a straightforward compression of geographic data while separately managing spatial indexing in some fashion that's applicable to that specific situation. For example cardboard.
I'm not opposed to a more compressed serialization of indexes, but my first reaction is that it is a separate concern that shouldn't be addressed in geobuf.
A separate repo for rbush-based spatial indexing & serialization sounds like good separation of concerns.
@mourner My concern with serializing the R-Tree vs bounding boxes or other spatial index precursors was simply that the other implementations wouldn't be able to consume the serialized R-Tree directly, their APIs would probably expect the index precursors (e.g., bboxes) instead. But I'm outside my domain on the internals of spatial indexes, so I would definitely defer to you on this. rbush is awesome and would serve as a good implementation to build this around.
@brendan-ward I don't know much about the cases involving serialized spatial indexes and what kind of applications would consume it, but I don't see much point in storing an index if it can't actually be used as an index.
E.g. to create an index from bboxes only, you would spend time running R-tree indexing on them after unpacking, which kinda defeats the purpose of having a stored precalculated index. And if you're fine with just bboxes, you could just as well precalculate and put them in the input data (GeoJSON or TopoJSON) since they're in the spec, and then store in Geobuf without any index extensions.
Also, the main challenge of spatial indexing is creating the index. Using it is dead-simple (like, simple depth-first search through a tree is less than 10 lines of code).
@mourner: very interesting work! We have been missing an efficient modern geo format for a long time. Some ideas:
I like the idea of (optionally) storing bboxes with features. It would allow quickly reading a file picking out only those features inside a given bbox. So even without an index it could allow faster access to some features without the overhead of de-serializing everything. Not sure how useful and how fast it would be in the real world. But reading data sequentially can often be faster than jumping through indexes.
Have you looked at the OSM PBF format? It solves some of the same problems for OSM and also uses Protobuf. What I especially like about it, is its block structure. Data is grouped into blocks and each block is encoded on its own. PBF has the same kind of lookup table for the strings as you have (for tags, usernames, etc. in PBFs case). But the lookup table is stored in each block. This allows easier streaming read and write because you don't have to wait for all the data to be complete before you can write the string table. It also allows for rather trivial parallelization of parsing because each block can be (de)serialized by itself in a different thread. This is what I do in Osmium. Of course it creates some overhead...
Protobuf (de)serialization has quite some overhead in my experience even in C++ due to the need for memory allocation. Your design of Multipolygons as repeated MultiLineStrings which are repeated LineStrings which are repeated coordinates looks straightforward, but might be expensive to encode and decode because of the nested messages. Maybe one large coordinate array with length fields telling us where sub-features start and end is better? This would need, of course, some experimentation.
The format should have some kind of versioning support and maybe some kind of header for optional features. Again, the OSM PBF format has an interesting (although not perfect) approach. It is important that features and settings (such as precision etc.) are located at the beginning of the file so they can be easily read before the rest of the file is parsed.
These are just a few ideas from the top of my head. I'd love to try a C++ implementation to see how things would be implemented there and how performance looks.
But reading data sequentially can often be faster than jumping through indexes.
Here's what I have in mind for the index: it would be very small compared to the data file, because it would only contain bboxes, without geometry or properties, but additionally storing the byte index to jump to for reading a particular feature from the main data file. Then to read the features you need, you would do a simple search through the index, save all the feature byte indexes and then just run through them. If non-sequential reads are a concern, you can just sort the indexes before reading. It seems that it would definitely be faster than reading through every feature. What do you think?
Regarding bboxes for every feature/geometry — it's pretty easy to add as an option, so not a problem.
PBF has the same kind of lookup table for the strings as you have (for tags, usernames, etc. in PBFs case). But the lookup table is stored in each block. This allows easier streaming read and write [...]
I see what you mean. I think we can find a compromise on this. Maybe store the unique keys globally — are usually the same through the whole data and are jut repeated, so the array will be small, and it will be very fast to read at the beginning. But store values per feature — those aren't duplicated as often as keys.
Protobuf (de)serialization has quite some overhead in my experience even in C++ due to the need for memory allocation.
Protobuf 3 is going to solve this with arena allocation — it will know the deserialized size beforehand and will be able to preallocate all the memory needed.
Your design of Multipolygons as repeated MultiLineStrings which are repeated LineStrings which are repeated coordinates looks straightforward, but might be expensive to encode and decode because of the nested messages. Maybe one large coordinate array with length fields telling us where sub-features start and end is better?
This sounds like a great idea. I'll definitely give this a try — sounds pretty easy to implement. So I'll just have an array of coordinates and an array of lengths, e.g. for MultiPolygon:
@mourner: Byte indexes are problematic, because the protobuf library hides that from you. Maybe there is a way to get the byte index of a feature from the protobuf lib, but I doubt that. If you are going that route you'd have to serialize each feature separately using protobuf but then write each feature into the file yourself, probably with a length header.
...store the unique keys globally...
That sounds reasonable. As long as we keep the streaming capability, ie we are sure we know before we start writing all strings that will end up in the header.
Protobuf 3 is going to solve this with arena allocation
Well, the point is we don't want to allocate memory at all if we can avoid it. Don't know what Protobuf wants to do about it. Couldn't find anything about Protobuf 3. Do you have a link?
What would you use for the decoder? Our mapbox/pbf.hpp?
In the first version the official protobuf lib. Maybe later something else. When I tried the mapbox pbf parser the last time for Osmium, it didn't implement everything in the Protobuf spec and couldn't do everything that OSM PBF needed. Have to re-evaluate this at some point and see whether it would work for geobuf.
Big +1 behind giving a shot to @joto's idea about Maybe one large coordinate array with length fields telling us where sub-features start and end is better? This would need, of course, some experimentation. (refs springmeyer/geojsonp#2 (comment)). The goal would be to allow [ packed = true ]; for the entire geometry since this would be potentially much more compact.
Also I agree that while arena allocation in proto3 might mitigate the cost of memory allocation for parsing the message into protobuf objects the ideal in high performance situations would be to avoid needing to allocate at all and instead go directly from messages -> your applications objects (skipping the intermediate objects libprotobuf creates). This is what https://github.com/mapbox/pbf.hpp and https://github.com/mapbox/pbf are about. I think you all understand this but wanting to re-state for clarity.
@joto about byte indexes — there can be a problem if you use the official protobuf library which is very high-level. But with a more low-level approach like pbf.hpp there should be no problem keeping track of byte indexes, and no problem serializing everything in one pass without any additional work.
When I tried the mapbox pbf parser the last time for Osmium, it didn't implement everything in the Protobuf spec
Lets create issues on any things lacking there. It's simple enough to add something relatively quickly, and there's @kkaefer to help.
@springmeyer implemented flattened array encoding — it doesn't make things much more compact (just a bit of bytes here and there) because most of the weight is still linestrings which are packed — not as many multilinestrings and multipolygons in real world datasets, and they don't have a lot of rings usually — just a couple.
But what I like about this approach is simplicity. It makes both schema and encoding/decoding code simpler.
@mourner: re byte indexes: I think it is important to stay compatible with the official protobuf libraries. If nothing else it allows other people to have their own implementations based on those libraries possibly in many different languages.
@joto good point. I'll think about this. At least we can do indexing + byte indexes as a separate tool, not baking anything specific to it into the Geobuf schema itself.
all geometries are now encoded as a flat packed coord array + flat packed array of lengths used to derive MultiLineString/Polygon/MultiPolygon structure if necessary — shaves off a bit of bytes and potentially makes encoding/decoding faster
all values are now encoded in each feature as opposed to global values array, to make incremental parsing easier and faster — this doesn't lead to a noticeable size increase since repeated values are not that common
Geobuf now encodes any arbitrary properties of any objects as well, making it possible to store GeoJSON/TopoJSON extensions and custom stuff
Activity
springmeyer commentedon Dec 11, 2014
Awesome to learn about oneof. Overall your ideas look excellent. I'll comment more when I've had time to digest the proposed schema.
brendan-ward commentedon Dec 11, 2014
This looks like a great direction.
What about letting the feature id field leverage
oneofto allow encoding asuint32values (much more common than string IDs in many cases we deal with):mourner commentedon Dec 11, 2014
@brendan-ward yes, excellent suggestion! Added it to the proto gist, except including oneof directly in the Feature message, since oneof fields are already optional by nature.
mourner commentedon Dec 11, 2014
also cc @mick @rclark @sgillies in case you're interested
mourner commentedon Dec 11, 2014
OK guys, today I wrote the encoder of my proposed format in Python to test it out, and the results are pretty mind-blowing.
Here's the comparison for the 100Mb 5.4-million points (1e4 precision) US zip codes GeoJSON (that I used for testing geojson-vt):
Here's another sample, Idaho from geojson-fixtures:
The keys of properties stored are unique. I think I can squish out just a bit more by making string values unique too.
tmcw commentedon Dec 11, 2014
Cool. To be clear, "your geobuf" can be "our geobuf" :)
mourner commentedon Dec 11, 2014
@tmcw sure, I just wanted to share the motivation before suggesting that I completely rewrite this repo from scratch while breaking compatibility, which can be very annoying. :)
tmcw commentedon Dec 11, 2014
I don't mind anything being rewritten ever as long as it's better :) But the people who might feel the pain of this would be @mick basically, because it's used in cardboard afaik
mick commentedon Dec 11, 2014
@mourner I will feel the pain. But that's is ok, this like a great rewrite. I can migrate my usage of geobuf to the new version.
mourner commentedon Dec 12, 2014
Reference Python encoder here: https://github.com/mapbox/pygeobuf
Working on decoder and some tests, and after that we can start reworking the JS implementation.
kkaefer commentedon Dec 12, 2014
Any comparisons to TopoJSON?
mourner commentedon Dec 12, 2014
@kkaefer depends on the data — Geobuf is a GeoJSON equivalent, without encoding topology. It will probably be smaller compared to TopoJSON, but I'm thinking about making it support TopoJSON encoding as well for even more compression (or making a separate protobuf schema for TopoJSON).
16 remaining items
mourner commentedon Dec 16, 2014
@brendan-ward btw, numbers for the Idaho TopoJSON test case above:
rclark commentedon Dec 16, 2014
I would just chime in to say that I imagine many implementations of geobuf will use it as a straightforward compression of geographic data while separately managing spatial indexing in some fashion that's applicable to that specific situation. For example cardboard.
I'm not opposed to a more compressed serialization of indexes, but my first reaction is that it is a separate concern that shouldn't be addressed in geobuf.
mourner commentedon Dec 16, 2014
@rclark 👍 to a separate repo for rbush-based spatial indexing and compressed index serialization.
brendan-ward commentedon Dec 16, 2014
A separate repo for rbush-based spatial indexing & serialization sounds like good separation of concerns.
@mourner My concern with serializing the R-Tree vs bounding boxes or other spatial index precursors was simply that the other implementations wouldn't be able to consume the serialized R-Tree directly, their APIs would probably expect the index precursors (e.g., bboxes) instead. But I'm outside my domain on the internals of spatial indexes, so I would definitely defer to you on this. rbush is awesome and would serve as a good implementation to build this around.
mourner commentedon Dec 17, 2014
@brendan-ward I don't know much about the cases involving serialized spatial indexes and what kind of applications would consume it, but I don't see much point in storing an index if it can't actually be used as an index.
E.g. to create an index from bboxes only, you would spend time running R-tree indexing on them after unpacking, which kinda defeats the purpose of having a stored precalculated index. And if you're fine with just bboxes, you could just as well precalculate and put them in the input data (GeoJSON or TopoJSON) since they're in the spec, and then store in Geobuf without any index extensions.
Also, the main challenge of spatial indexing is creating the index. Using it is dead-simple (like, simple depth-first search through a tree is less than 10 lines of code).
joto commentedon Dec 19, 2014
@mourner: very interesting work! We have been missing an efficient modern geo format for a long time. Some ideas:
These are just a few ideas from the top of my head. I'd love to try a C++ implementation to see how things would be implemented there and how performance looks.
mourner commentedon Dec 19, 2014
@joto thanks for the awesome feedback!
Here's what I have in mind for the index: it would be very small compared to the data file, because it would only contain bboxes, without geometry or properties, but additionally storing the byte index to jump to for reading a particular feature from the main data file. Then to read the features you need, you would do a simple search through the index, save all the feature byte indexes and then just run through them. If non-sequential reads are a concern, you can just sort the indexes before reading. It seems that it would definitely be faster than reading through every feature. What do you think?
Regarding bboxes for every feature/geometry — it's pretty easy to add as an option, so not a problem.
I see what you mean. I think we can find a compromise on this. Maybe store the unique keys globally — are usually the same through the whole data and are jut repeated, so the array will be small, and it will be very fast to read at the beginning. But store values per feature — those aren't duplicated as often as keys.
Protobuf 3 is going to solve this with arena allocation — it will know the deserialized size beforehand and will be able to preallocate all the memory needed.
This sounds like a great idea. I'll definitely give this a try — sounds pretty easy to implement. So I'll just have an array of coordinates and an array of lengths, e.g. for MultiPolygon:
I'll also take a closer look at the OSM PBF schema.
What would you use for the decoder? Our mapbox/pbf.hpp?
joto commentedon Dec 19, 2014
@mourner: Byte indexes are problematic, because the protobuf library hides that from you. Maybe there is a way to get the byte index of a feature from the protobuf lib, but I doubt that. If you are going that route you'd have to serialize each feature separately using protobuf but then write each feature into the file yourself, probably with a length header.
That sounds reasonable. As long as we keep the streaming capability, ie we are sure we know before we start writing all strings that will end up in the header.
Well, the point is we don't want to allocate memory at all if we can avoid it. Don't know what Protobuf wants to do about it. Couldn't find anything about Protobuf 3. Do you have a link?
In the first version the official protobuf lib. Maybe later something else. When I tried the mapbox pbf parser the last time for Osmium, it didn't implement everything in the Protobuf spec and couldn't do everything that OSM PBF needed. Have to re-evaluate this at some point and see whether it would work for geobuf.
springmeyer commentedon Dec 19, 2014
Big +1 behind giving a shot to @joto's idea about
Maybe one large coordinate array with length fields telling us where sub-features start and end is better? This would need, of course, some experimentation.(refs springmeyer/geojsonp#2 (comment)). The goal would be to allow[ packed = true ];for the entire geometry since this would be potentially much more compact.Also I agree that while arena allocation in proto3 might mitigate the cost of memory allocation for parsing the message into protobuf objects the ideal in high performance situations would be to avoid needing to allocate at all and instead go directly from messages -> your applications objects (skipping the intermediate objects libprotobuf creates). This is what https://github.com/mapbox/pbf.hpp and https://github.com/mapbox/pbf are about. I think you all understand this but wanting to re-state for clarity.
mourner commentedon Dec 20, 2014
@joto about byte indexes — there can be a problem if you use the official protobuf library which is very high-level. But with a more low-level approach like pbf.hpp there should be no problem keeping track of byte indexes, and no problem serializing everything in one pass without any additional work.
Lets create issues on any things lacking there. It's simple enough to add something relatively quickly, and there's @kkaefer to help.
@springmeyer implemented flattened array encoding — it doesn't make things much more compact (just a bit of bytes here and there) because most of the weight is still linestrings which are packed — not as many multilinestrings and multipolygons in real world datasets, and they don't have a lot of rings usually — just a couple.
But what I like about this approach is simplicity. It makes both schema and encoding/decoding code simpler.
joto commentedon Dec 20, 2014
@mourner: re byte indexes: I think it is important to stay compatible with the official protobuf libraries. If nothing else it allows other people to have their own implementations based on those libraries possibly in many different languages.
mourner commentedon Dec 20, 2014
@joto good point. I'll think about this. At least we can do indexing + byte indexes as a separate tool, not baking anything specific to it into the Geobuf schema itself.
mourner commentedon Dec 23, 2014
Made some progress:
mourner commentedon Jan 9, 2015
JS encoder/decoder implemented in #29