A walk on the wild side

January 28th, 2012

Lou Reed, topological versionI’ve been spending the last few days profiling and optimizing the new simple-to-topological converter you will find in PostGIS 2.0.0 thanks to a community effort.

The most expensive operation was found to be the ST_AddEdgeModFace function, which adds an edge and checks if such edge creates a new face.

Face-splitting detection was implemented using a brute force approach consisting in invoking the GEOS polygonizer and then checking if any polygon created contained the newly added edge in its boundary. It can get pretty wild when you add an edge in the universe face, and thus end up feeding the polygonizer with thousands of edges.

So I started thinking about two fields we make great effort to maintain and were (so far) unused: next_left_edge and next_right_edge. Each edge has these two pointers which may be used to walk clockwise along its left or right face.

By walking on the edge side you can know a lot about the edge you add !

First of all if you get to the other side of the edge you know for sure you didn’t split any face (no ring was created). Second, you can easily get the identifiers of all the edges you’ll need to update for setting their new left_face and right_face attributes. Finally by computing ring winding you can know whether each side of the edge is forming a fill or an hole.

I was afraid that such approach could have been slower than the brute force one, mainly due to more database querying and IO. But the dataset I was using for a testcase (Italian municipalities by ISTAT: 8094 multipolygons with an average of 560 vertices each) showed a speedup of up to 10x !

The testsuite was an invaluable tool for correctness checking. I actually spent a fair amount of time enhancing it further with new corner cases (new algorithm, new corners!).

I’m still checking correctness of  the resulting topology for the ISTAT case,  making sure that the improvements didn’t introduce any regression. Hopefully next week the code will be committed upstream for broader delectation. Stay tuned, prepare your input for conversion and start playing and timing the current routines !

GEOS 3.3.2 released

January 6th, 2012

The second bugfix release in the 3.3 branch of GEOS was released today.

This is the version required by the topology support shipped with the upcoming PostGIS 2.0 release.
Everyone is recommended to upgrade. Changes can be read here, package can be downloaded here.

Topology cleaning with PostGIS

November 21st, 2011

An early tester of the new PostGIS Topology submitted an interesting dataset which kept me busy for a couple of weeks fixing a bunch of bugs related to numerical stability/robustness.

Finally, the ST_CreateTopoGeo function succeeded and imported the dataset as a proper topological schema. Here’s what it looks like:

Edges of the built topology

At a first glance it doesn’t seem to be particularly problematic. Here’s the composition summary:

=# select topologysummary('small_sample_topo');
                    topologysummary
--------------------------------------------------------
 Topology small_sample_topo (2042), SRID 0, precision 0
 83 nodes, 156 edges, 74 faces, 0 topogeoms in 0 layers

But the devil hides at high zoom levels. Where to zoom ? What are we looking for ?

We are guaranteed none of the constructed edges cross so the only leftover problem we might encounter is very small faces constructed wherever the original input had small overlaps or underlaps (gaps). We can have a visual signal of those faces by creating a view showing faces with an area below a given threshold. Let’s do that:

CREATE VIEW small_sample_topo.small_areas AS
SELECT face_id, st_getfacegeometry('small_sample_topo', face_id)
FROM small_sample_topo.face
WHERE face_id > 0
AND st_area(st_getfacegeometry('small_sample_topo', face_id)) < 0.1;

That query would let us see where to find faces with area < 0.1 units. And here’s qgis showing it:

Areas smaller than 0.1 square units

Now we know where to zoom, and also the ID of the offending faces.

Let’s zoom in and show some labels:

Detail of small area

You can now see that face 59 is bound by (among others) edges 130 and 129. Just get rid of one of them to assign the area to an adjacent face.

We drop edge 130 using ST_RemEdgeModFace, assigning the area to face 52. Here’s the result:

Area after cleanup

Cleaning further would require removing further edges and thus getting rid of all the small faces. There’s a lot of room for automating such processes. The good new is you can now build your own automation around your specific use cases while still using robust and standard foundations.

It is to be noted that the whole process I described here only involved the geometrical/topological level and didn’t affect at all the semantic/feature level. If we had TopoGeometry objects defined by the faces we’d also know which small faces were part of overlaps or underlaps and could then act consequently by adding or removing faces to the definition of the appropriate TopoGeometry object. Such step would have been required for the overlap situations as the ST_RemEdgeModFace function doesn’t let you change the shape of a defined TopoGeometry.

Unfortunately the semantic level is lost when using the ISO functions, as the whole ISO topology model doesn’t deal with features at all. This is why I think PostGIS would benefit from having a function that converts your simple features into topologically-defined features by adding any missing primitive to a topology schema and constructing the feature for you. Such function, is only waiting for sponsorship to become a reality of PostGIS 2.0. If you like what we’re building for your data integrity, please consider supporting the effort!

Ming 0.4.4 released

October 26th, 2011

I’m happy to announce that release 0.4.4 of the Ming library is out.

Ming is an SWF output library with binding for C, C++, PHP, Perl, Python and more.

It’s stable, alive and waiting for you at his new github location.

Don’t let Flash get in your way, do your part for an open web!

Changes in this release:

  • Generally improve swftoscript and decompiler
  • Change makefdb to name output files by font ID, to play nicer with swftoscript.
  • Add support for ‘class A extends B’ syntax in actioncompiler
  • Fix bug in ‘makeswf’ failing to catch some compile errors (bugzilla #94) and being too silent in swf embedding errors
  • Fix bug in action compiler dealing with class methods (bugzilla #94)
  • Add support for libpng > 1.4 (bugzilla #96)
  • Add font kernings support (bugzilla #95)
  • Add button characters export capabilities
  • Add support for ‘swfAction <code>’ syntax in asm blocks

PostGIS topology ISO SQL/MM complete

October 14th, 2011

PostGIS implementation of the ISO SQL/MM Topology-Geometry model is finally complete.

The SQL/MM model is just a portion of the whole topology support, but an important one, including schema definition and functions to create and populate the schema with primitive components (nodes, edges, faces).

In addition to the base model, PostGIS adds a TopoGeometry data type for use wherever you would normally use a Geometry type, except the former will be defined by references to primitive, shared, topological components.

The work on SQL/MM topology for PostGIS 2.0 was mainly funded by the Land Management department of the Tuscany Region as part of a larger project aimed at enhancing effectiveness of free software tools already in use within the institution for the management of the topographic database (chapeau!).

But more work is needed to make it easier to convert your layers to topological layers, your Geometries to TopoGeometries. Because I know you want to generalize your vectors in a topologically consistent way, don’t you ? And you want to edit the boundaries just once. And do you want to waste all that space for your triangles ?

Feature freeze for 2.0 will be by the end of November. Drop a note ASAP if you’d like to help with getting the enhanced TopoGeometry constructor shipped with it.

GEOS 3.3.1 released

September 27th, 2011

The first bugfix release in the 3.3 branch of GEOS was released today.

Anyone running 3.3.0 is recommended to upgrade. Changes can be read here, package can be downloaded here.

By the way… did you try the PHP binding yet ? Configure with --enable-php to take a look !

Hack fever 2 : super mario sound

September 7th, 2011

I was supposed to leave for holidays on August 1st, but a summer fever held me back.

As it became traditional, I took the chance to do some Gnash hacking, approaching a bug that made playing Super Mario Bros unpleasant. The mario sound didn’t stop on game over thus overlapping with the new sound started for the new game. Annoying.

Super Mario

After some research the help came from Jan Flanders, an usual attender of the #gnash IRC channel. He pointed out that calling .stop() against an unattached Sound object should stop all playing event sounds. It took a couple of days from there to an automated testcase and a fix.

You can now Enjoy Your Mushrooms !

Getting just the tip of a remote git branch

June 7th, 2011

As projects move their code under git control, people get frustrated about being unable to do most basic operations they are used to perform with SVN or CVS. That’s a fact, so let’s see if I can relief some pain by sharing what I know or learn as I crawl the learning curve myself.

Yesterday I’ve met with Markus Neteler and he was complaining about being unable to checkout the release branch of QuantumGIS without filling up his laptop hard drive. He got me curious, so here are some numbers and a recipe.

An SVN checkout as of April 30  is 281 Mb in size, 1 of which being the .svn directory.

A full git clone at time of writing (June 7) is 330 Mb  in size, 200 of which being the .git database and  130 being the working copy (the “checkout”).

A full git clone contains all the data available in the original repository. Once you get the clone, you have all the branches and all the history. no need for any more bandwidth.

But Markus was only interested in a single branch, not the whole set, and he wanted no history either. So he could cloned just the objects referenced by the commit known as the release-1_7_0 branch and no further parents (back history). Here’s how you do:

git clone –depth 1 –branch release-1_7_0 git://github.com/qgis/Quantum-GIS.git

The resulting shallow repository (the .git directory) is 110 Mb in size. Add 133 Mb of working directory (yes, release-1_7_0 is 3 Mb bigger than master) for a total of 243 Mb disk space used.

NOTES:
  1. A shallow repository (one with short history) cannot be further cloned, but here are no problems pulling updates from the origin nor producing patches or pushing changes.
  2. If you don’t know in advance the name of the branch you can query it from the remote repository using git ls-remote
  3. Every git command has a manual page in the form: git-command (ie: man git-ls-remote)
Happy learning !

GEOS 3.3.0

May 31st, 2011

GEOS 3.3.0 is out: http://download.osgeo.org/geos/geos-3.3.0.tar.bz2

This release introduces a fair amount of new C-API interfaces and a brand new PHP binding. Full details in the NEWS file: http://trac.osgeo.org/geos/browser/tags/3.3.0/NEWS

As with any release since 3.0.0 there is complete binary compatibility with clients linked against the C-API. These include, but are not limited to, PostGIS. For a list of known clients: http://trac.osgeo.org/geos/wiki/Applications (add yours, if not already listed!)

GEOS is a C++ port of the JTS Topology Suite. This release targets version 1.12 of the library, but doesn’t reach full feature parity yet. Missing JTS functionalities:

  • Densifier class
  • Geometric similarity detection package (HausdorffSimilarityMeasure, AreaSimilarityMeasure)
  • MinimumDiameter.getMiminumRectangle()
  • Triangulation API
  • VoronoiDiagramBuilder
  • createSquircle and createSuperCircle in GeometricShapeFactory
  • MinimumClearance class
  • nearestNeighbours method to STRtree
  • RandomPointsBuilder / RandomPointsInGridBuilder
  • KochSnowflakeBuilder
  • SierpinksiCarpetBuilder

If you’d like to sponsor development of any of the above items (or others) for next feature release of GEOS (3.4.0) please drop me a note.

Software libero e tecnologie partecipative

May 30th, 2011

Lunedi’ 6 Giugno 2011 alle 20:30 presso la sede dell’ associazione Cubibi’ di Ispra si terra’ un’ incontro aperto sul software libero e le tecnologie partecipative.

[ via Madonnina del Grappa, 40-48 - Ispra. 6 Giugno 2011 ore 20:30 ]

L’ incontro e’ rivolto a tutti coloro che coltivano la propria curiosita’ come una risorsa e vogliono scoprire qualcosa in piu’ sugli strumenti elettronici che mediano sempre piu’ le nostre attivita’ personali e sociali estendendone o limitandone le possibilita’, aiutando od ostacolando la nostra capacita’ di scelta.

UPDATE: qui il materiale per stampare il volantino.