Skip to content

A revised breaking protobuf schema for Geobuf #27

Closed

Description

@mourner

The more I find ideas to improve the sizes of geobuf, the more I realize that we would need to completely rewrite the schema to support the improvements, breaking compatibility. So I'm opening the ticket to start a discussion about what a perfect Geobuf schema would look like (not to say this is a priority, but still a good thing to discuss).

I wrote a prototype schema with all the improvements I could think of here: https://gist.github.com/mourner/3c6ddca04c9772593302

The main difference is utilizing the power and flexibility of the new oneof statement to create a better and more compact schema.

Features:

  • the data itself contains information whether it's Feature, FeatureCollection, Geometry or GeometryCollection, so you don't need to guess this when decoding
  • keys and values for properties are stored separately in the top-level Data object, and features only store indexes to them (like vector-tile-spec does); keys and possibly values are reduced to unique values Fancy encoding for properties #26 — for much better properties packing
  • geometry coordinates are a oneof set of different fields (depending on type), which solves the ambiguity with null values since a oneof field can be empty (without a default value), and also makes it easier to understand and work with
  • feature has a oneof of either geometry or geometry collection instead of repeated geometries
  • coordinates are stored as delta-encoded sint32 to take full advantage of varint and zig-zag encoding Switch to sint32 for coords #24 — for much more compact geometries
  • Value message is also a oneof of different value types
  • the property value int type is sint32 instead of int64 — it's more compact and JS doesn't actually handle int64 values; in addition, uint type becomes uint32
  • the data contains optional flag indicating whether it contains altitude (third coordinate), since this is a global-level setting
  • the data also contains optional precision information (6 by default) Configurable precision #25

@tmcw @springmeyer what do you think?

Activity

  1. self-assigned this
    on Dec 10, 2014
  2. springmeyer commented on Dec 11, 2014

    @springmeyer

    Awesome to learn about oneof. Overall your ideas look excellent. I'll comment more when I've had time to digest the proposed schema.

  3. brendan-ward commented on Dec 11, 2014

    @brendan-ward

    This looks like a great direction.

    What about letting the feature id field leverage oneof to allow encoding as uint32 values (much more common than string IDs in many cases we deal with):

    message Feature {
        oneof geometry_type {
            Geometry geometry = 1;
            GeometryCollection geometry_collection = 2;
        }
        repeated uint32 properties = 3 [packed = true];
        optional ID id = 4;
    }
    
    message ID {
        oneof value {
            string string_value = 1;
            uint32 uint_value = 2;
        }
    }
    
  4. mourner commented on Dec 11, 2014

    @mourner
    MemberAuthor

    @brendan-ward yes, excellent suggestion! Added it to the proto gist, except including oneof directly in the Feature message, since oneof fields are already optional by nature.

  5. mourner commented on Dec 11, 2014

    @mourner
    MemberAuthor

    also cc @mick @rclark @sgillies in case you're interested

  6. mourner commented on Dec 11, 2014

    @mourner
    MemberAuthor

    OK guys, today I wrote the encoder of my proposed format in Python to test it out, and the results are pretty mind-blowing.

    Here's the comparison for the 100Mb 5.4-million points (1e4 precision) US zip codes GeoJSON (that I used for testing geojson-vt):

    normal gzipped
    GeoJSON 101.85 MB 26.67 MB
    current geobuf 73.53 MB 31.09 MB
    my geobuf 1e6 22.27 MB 16.39 MB
    my geobuf 1e4 12.43 MB 10.62 MB

    Here's another sample, Idaho from geojson-fixtures:

    normal gzipped
    GeoJSON 10.93 MB 2.58 MB
    current geobuf 5.53 MB 2.07 MB
    my geobuf 1e6 1.38 MB 1.18 MB

    The keys of properties stored are unique. I think I can squish out just a bit more by making string values unique too.

  7. tmcw commented on Dec 11, 2014

    @tmcw
    Contributor

    Cool. To be clear, "your geobuf" can be "our geobuf" :)

  8. reopened this on Dec 11, 2014
  9. mourner commented on Dec 11, 2014

    @mourner
    MemberAuthor

    @tmcw sure, I just wanted to share the motivation before suggesting that I completely rewrite this repo from scratch while breaking compatibility, which can be very annoying. :)

  10. tmcw commented on Dec 11, 2014

    @tmcw
    Contributor

    I don't mind anything being rewritten ever as long as it's better :) But the people who might feel the pain of this would be @mick basically, because it's used in cardboard afaik

  11. mick commented on Dec 11, 2014

    @mick
    Contributor

    @mourner I will feel the pain. But that's is ok, this like a great rewrite. I can migrate my usage of geobuf to the new version.

  12. mourner commented on Dec 12, 2014

    @mourner
    MemberAuthor

    Reference Python encoder here: https://github.com/mapbox/pygeobuf
    Working on decoder and some tests, and after that we can start reworking the JS implementation.

  13. kkaefer commented on Dec 12, 2014

    @kkaefer
    Member

    Any comparisons to TopoJSON?

  14. mourner commented on Dec 12, 2014

    @mourner
    MemberAuthor

    @kkaefer depends on the data — Geobuf is a GeoJSON equivalent, without encoding topology. It will probably be smaller compared to TopoJSON, but I'm thinking about making it support TopoJSON encoding as well for even more compression (or making a separate protobuf schema for TopoJSON).

  15. 16 remaining items

  16. mourner commented on Dec 16, 2014

    @mourner
    MemberAuthor

    @brendan-ward btw, numbers for the Idaho TopoJSON test case above:

    normal gzipped
    TopoJSON 1e6 1.95 MB 560 KB
    Geobuf 545 KB 451 KB
  17. rclark commented on Dec 16, 2014

    @rclark
    Contributor

    I would just chime in to say that I imagine many implementations of geobuf will use it as a straightforward compression of geographic data while separately managing spatial indexing in some fashion that's applicable to that specific situation. For example cardboard.

    I'm not opposed to a more compressed serialization of indexes, but my first reaction is that it is a separate concern that shouldn't be addressed in geobuf.

  18. mourner commented on Dec 16, 2014

    @mourner
    MemberAuthor

    @rclark 👍 to a separate repo for rbush-based spatial indexing and compressed index serialization.

  19. brendan-ward commented on Dec 16, 2014

    @brendan-ward

    A separate repo for rbush-based spatial indexing & serialization sounds like good separation of concerns.

    @mourner My concern with serializing the R-Tree vs bounding boxes or other spatial index precursors was simply that the other implementations wouldn't be able to consume the serialized R-Tree directly, their APIs would probably expect the index precursors (e.g., bboxes) instead. But I'm outside my domain on the internals of spatial indexes, so I would definitely defer to you on this. rbush is awesome and would serve as a good implementation to build this around.

  20. mourner commented on Dec 17, 2014

    @mourner
    MemberAuthor

    @brendan-ward I don't know much about the cases involving serialized spatial indexes and what kind of applications would consume it, but I don't see much point in storing an index if it can't actually be used as an index.

    E.g. to create an index from bboxes only, you would spend time running R-tree indexing on them after unpacking, which kinda defeats the purpose of having a stored precalculated index. And if you're fine with just bboxes, you could just as well precalculate and put them in the input data (GeoJSON or TopoJSON) since they're in the spec, and then store in Geobuf without any index extensions.

    Also, the main challenge of spatial indexing is creating the index. Using it is dead-simple (like, simple depth-first search through a tree is less than 10 lines of code).

  21. joto commented on Dec 19, 2014

    @joto

    @mourner: very interesting work! We have been missing an efficient modern geo format for a long time. Some ideas:

    • I like the idea of (optionally) storing bboxes with features. It would allow quickly reading a file picking out only those features inside a given bbox. So even without an index it could allow faster access to some features without the overhead of de-serializing everything. Not sure how useful and how fast it would be in the real world. But reading data sequentially can often be faster than jumping through indexes.
    • Have you looked at the OSM PBF format? It solves some of the same problems for OSM and also uses Protobuf. What I especially like about it, is its block structure. Data is grouped into blocks and each block is encoded on its own. PBF has the same kind of lookup table for the strings as you have (for tags, usernames, etc. in PBFs case). But the lookup table is stored in each block. This allows easier streaming read and write because you don't have to wait for all the data to be complete before you can write the string table. It also allows for rather trivial parallelization of parsing because each block can be (de)serialized by itself in a different thread. This is what I do in Osmium. Of course it creates some overhead...
    • Protobuf (de)serialization has quite some overhead in my experience even in C++ due to the need for memory allocation. Your design of Multipolygons as repeated MultiLineStrings which are repeated LineStrings which are repeated coordinates looks straightforward, but might be expensive to encode and decode because of the nested messages. Maybe one large coordinate array with length fields telling us where sub-features start and end is better? This would need, of course, some experimentation.
    • The format should have some kind of versioning support and maybe some kind of header for optional features. Again, the OSM PBF format has an interesting (although not perfect) approach. It is important that features and settings (such as precision etc.) are located at the beginning of the file so they can be easily read before the rest of the file is parsed.

    These are just a few ideas from the top of my head. I'd love to try a C++ implementation to see how things would be implemented there and how performance looks.

  22. mourner commented on Dec 19, 2014

    @mourner
    MemberAuthor

    @joto thanks for the awesome feedback!

    But reading data sequentially can often be faster than jumping through indexes.

    Here's what I have in mind for the index: it would be very small compared to the data file, because it would only contain bboxes, without geometry or properties, but additionally storing the byte index to jump to for reading a particular feature from the main data file. Then to read the features you need, you would do a simple search through the index, save all the feature byte indexes and then just run through them. If non-sequential reads are a concern, you can just sort the indexes before reading. It seems that it would definitely be faster than reading through every feature. What do you think?

    Regarding bboxes for every feature/geometry — it's pretty easy to add as an option, so not a problem.

    PBF has the same kind of lookup table for the strings as you have (for tags, usernames, etc. in PBFs case). But the lookup table is stored in each block. This allows easier streaming read and write [...]

    I see what you mean. I think we can find a compromise on this. Maybe store the unique keys globally — are usually the same through the whole data and are jut repeated, so the array will be small, and it will be very fast to read at the beginning. But store values per feature — those aren't duplicated as often as keys.

    Protobuf (de)serialization has quite some overhead in my experience even in C++ due to the need for memory allocation.

    Protobuf 3 is going to solve this with arena allocation — it will know the deserialized size beforehand and will be able to preallocate all the memory needed.

    Your design of Multipolygons as repeated MultiLineStrings which are repeated LineStrings which are repeated coordinates looks straightforward, but might be expensive to encode and decode because of the nested messages. Maybe one large coordinate array with length fields telling us where sub-features start and end is better?

    This sounds like a great idea. I'll definitely give this a try — sounds pretty easy to implement. So I'll just have an array of coordinates and an array of lengths, e.g. for MultiPolygon:

    lengths: [polygons, poly1_rings, ring1, ring2, ..., poly2_rings, ring1, ring2, ...]
    

    I'll also take a closer look at the OSM PBF schema.

    I'd love to try a C++ implementation to see how things would be implemented there and how performance looks.

    What would you use for the decoder? Our mapbox/pbf.hpp?

  23. joto commented on Dec 19, 2014

    @joto

    @mourner: Byte indexes are problematic, because the protobuf library hides that from you. Maybe there is a way to get the byte index of a feature from the protobuf lib, but I doubt that. If you are going that route you'd have to serialize each feature separately using protobuf but then write each feature into the file yourself, probably with a length header.

    ...store the unique keys globally...

    That sounds reasonable. As long as we keep the streaming capability, ie we are sure we know before we start writing all strings that will end up in the header.

    Protobuf 3 is going to solve this with arena allocation

    Well, the point is we don't want to allocate memory at all if we can avoid it. Don't know what Protobuf wants to do about it. Couldn't find anything about Protobuf 3. Do you have a link?

    What would you use for the decoder? Our mapbox/pbf.hpp?

    In the first version the official protobuf lib. Maybe later something else. When I tried the mapbox pbf parser the last time for Osmium, it didn't implement everything in the Protobuf spec and couldn't do everything that OSM PBF needed. Have to re-evaluate this at some point and see whether it would work for geobuf.

  24. springmeyer commented on Dec 19, 2014

    @springmeyer

    Big +1 behind giving a shot to @joto's idea about Maybe one large coordinate array with length fields telling us where sub-features start and end is better? This would need, of course, some experimentation. (refs springmeyer/geojsonp#2 (comment)). The goal would be to allow [ packed = true ]; for the entire geometry since this would be potentially much more compact.

    Also I agree that while arena allocation in proto3 might mitigate the cost of memory allocation for parsing the message into protobuf objects the ideal in high performance situations would be to avoid needing to allocate at all and instead go directly from messages -> your applications objects (skipping the intermediate objects libprotobuf creates). This is what https://github.com/mapbox/pbf.hpp and https://github.com/mapbox/pbf are about. I think you all understand this but wanting to re-state for clarity.

  25. mourner commented on Dec 20, 2014

    @mourner
    MemberAuthor

    @joto about byte indexes — there can be a problem if you use the official protobuf library which is very high-level. But with a more low-level approach like pbf.hpp there should be no problem keeping track of byte indexes, and no problem serializing everything in one pass without any additional work.

    When I tried the mapbox pbf parser the last time for Osmium, it didn't implement everything in the Protobuf spec

    Lets create issues on any things lacking there. It's simple enough to add something relatively quickly, and there's @kkaefer to help.

    @springmeyer implemented flattened array encoding — it doesn't make things much more compact (just a bit of bytes here and there) because most of the weight is still linestrings which are packed — not as many multilinestrings and multipolygons in real world datasets, and they don't have a lot of rings usually — just a couple.

    But what I like about this approach is simplicity. It makes both schema and encoding/decoding code simpler.

  26. joto commented on Dec 20, 2014

    @joto

    @mourner: re byte indexes: I think it is important to stay compatible with the official protobuf libraries. If nothing else it allows other people to have their own implementations based on those libraries possibly in many different languages.

  27. mourner commented on Dec 20, 2014

    @mourner
    MemberAuthor

    @joto good point. I'll think about this. At least we can do indexing + byte indexes as a separate tool, not baking anything specific to it into the Geobuf schema itself.

  28. mourner commented on Dec 23, 2014

    @mourner
    MemberAuthor

    Made some progress:

    • all geometries are now encoded as a flat packed coord array + flat packed array of lengths used to derive MultiLineString/Polygon/MultiPolygon structure if necessary — shaves off a bit of bytes and potentially makes encoding/decoding faster
    • all values are now encoded in each feature as opposed to global values array, to make incremental parsing easier and faster — this doesn't lead to a noticeable size increase since repeated values are not that common
    • Geobuf now encodes any arbitrary properties of any objects as well, making it possible to store GeoJSON/TopoJSON extensions and custom stuff
  29. mourner commented on Jan 9, 2015

    @mourner
    MemberAuthor

    JS encoder/decoder implemented in #29

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions