[Community] EWKT/EWKB support in Shapely

Discussion:

Matthias Uden

2014-05-06 08:40:45 UTC

Hi list,

I'm new to Shapely and I just recognised, that exporting EWKT/EWKB is
not supported, only plain WKT/WKB. [1]
I'm not entirely sure but I think GEOS supports including the SRID? [2]

Are there any plans on supporting this feature in future Shapely releases?

AFAIK, the only other possibility to export EWKT/EWKB with Python would
then be to use the Django
GEOS binding, which supports it, cf. [3].
However, I'd like to avoid using Django because it is a bit of an
overhead for my project
since I'm not doing any web development but rather backend geodata
processing.

If anyone can think of other possibilities instead, please let me know.
Any help is appreciated, thanks a lot in advance!

Cheers,
Matthias

[1] - http://toblerity.org/shapely/manual.html#well-known-formats
[2] - http://geos.osgeo.org/doxygen/classgeos_1_1io_1_1WKBWriter.html
[3] -
https://docs.djangoproject.com/en/dev/ref/contrib/gis/geos/#django.contrib.gis.geos.WKBWriter

Sean Gillies

2014-05-06 14:02:27 UTC

Permalink

Hi Matthias,

There are no plans in Shapely for extended WKB or WKT. I recommend
inserting the SRID value into the text or byte strings from Shapely.
Something like

ewkt = ";".join(["SRID=4326", wkt])

On Tue, May 6, 2014 at 2:40 AM, Matthias Uden

Post by Matthias Uden
Hi list,
I'm new to Shapely and I just recognised, that exporting EWKT/EWKB is not
supported, only plain WKT/WKB. [1]
I'm not entirely sure but I think GEOS supports including the SRID? [2]
Are there any plans on supporting this feature in future Shapely releases?
AFAIK, the only other possibility to export EWKT/EWKB with Python would
then be to use the Django
GEOS binding, which supports it, cf. [3].
However, I'd like to avoid using Django because it is a bit of an overhead
for my project
since I'm not doing any web development but rather backend geodata
processing.
If anyone can think of other possibilities instead, please let me know.
Any help is appreciated, thanks a lot in advance!
Cheers,
Matthias
[1] - http://toblerity.org/shapely/manual.html#well-known-formats
[2] - http://geos.osgeo.org/doxygen/classgeos_1_1io_1_1WKBWriter.html
[3] - https://docs.djangoproject.com/en/dev/ref/contrib/gis/
geos/#django.contrib.gis.geos.WKBWriter
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

--
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20140506/7f618e16/attachment.htm>

Mike Toews

2014-05-06 23:15:30 UTC

Permalink

This post might be inappropriate. Click to display it.

Matthias Uden

2014-05-07 09:06:32 UTC

Permalink

Hi Mike & Sean,

thanks a lot for your quick help!
The proposed method using the direct GEOS call worked fine. I didn't
know I can modify the GEOS properties directly this way.

I'd suggest to include this in a little convenient wrapper method in
Shapely, as it should not be too hard to accomplish.
But using it this way is fine for me as well.

Thanks again and all the best,
Matthias

Post by Sean Gillies
Hi Matthias,
The Writers from GEOS already use the EWKB dialect (rather than the
ISO dialect) of WKB, however by default these outputs are turned off
to not include SRID and not output Z coordinates. More on that here
[1].
To hack shapely to include SRIDs, this is what to do: modify the
WKBWritier default to include SRID (default is False), and set the
SRID to the geometry using a GEOS call.
from shapely import geos
from shapely.geometry import Point
geos.WKBWriter.defaults['include_srid'] = True
p = Point(1,2)
print(p.wkb_hex)
geos.lgeos.GEOSSetSRID(p._geom, 4326)
print(p.wkb_hex)
This could be a simple feature request, where the setters and getters
to an 'srid' property on BaseGeometry are simply mapped to GEOSGetSRID
p.srid = 4326
As for the WKTWriter, there is no 'include_srid' option with the GEOS
C API, but this isn't too difficult to modify the Python class to
prepend 'SRID=4326' to the WKT.
-Mike
[1] http://gis.stackexchange.com/a/89388/1872

Post by Matthias Uden
Hi list,
I'm new to Shapely and I just recognised, that exporting EWKT/EWKB is not
supported, only plain WKT/WKB. [1]
I'm not entirely sure but I think GEOS supports including the SRID? [2]
Are there any plans on supporting this feature in future Shapely releases?
AFAIK, the only other possibility to export EWKT/EWKB with Python would then
be to use the Django
GEOS binding, which supports it, cf. [3].
However, I'd like to avoid using Django because it is a bit of an overhead
for my project
since I'm not doing any web development but rather backend geodata
processing.
If anyone can think of other possibilities instead, please let me know.
Any help is appreciated, thanks a lot in advance!
Cheers,
Matthias
[1] - http://toblerity.org/shapely/manual.html#well-known-formats
[2] - http://geos.osgeo.org/doxygen/classgeos_1_1io_1_1WKBWriter.html
[3] -
https://docs.djangoproject.com/en/dev/ref/contrib/gis/geos/#django.contrib.gis.geos.WKBWriter

_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

Michael Nolde

2014-05-09 12:14:27 UTC

Permalink

Hi there,

I noticed some strange behavior when reading float values from shapefile
attributes with fiona.
In the attribute table, I have the following values (type of column is
'float'):
1.5, 613.0, 0.9

but when I open the shapefile with fiona and read the attributes with
something like:

somevar = float(item['properties']['VAR'])
I get: 1.0, 613.0, 0.0

So the values seem to be rounded down. Am I missing something? Could
this be a bug?
The fiona version I use is: 1.1.4 (Python 2.7.6 on Ubuntu 14.04 64 bit).

Best regards,
Michael

--
Dr. rer. nat. Michael Nolde
Christian-Albrechts-Universit?t zu Kiel
Geographisches Institut (Sektion Geographie)
Neufeldtstra?e 10, Laurispark - Geb. 32, 24118 Kiel
Tel.: 0431 880-4909
E-Mail: nolde at geographie.uni-kiel.de
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20140509/6506befa/attachment.htm>

Sean Gillies

2014-05-09 19:32:04 UTC

Permalink

Hi Michael,

I can't reproduce this error with any data I have. Can you put your
shapefile up for download and tell me which field(s) are the ones being
rounded? I'll take a look if you can.

Yours,

On Fri, May 9, 2014 at 6:14 AM, Michael Nolde

Post by Michael Nolde
Hi there,
I noticed some strange behavior when reading float values from shapefile
attributes with fiona.
In the attribute table, I have the following values (type of column is
1.5, 613.0, 0.9
but when I open the shapefile with fiona and read the attributes with
somevar = float(item['properties']['VAR'])
I get: 1.0, 613.0, 0.0
So the values seem to be rounded down. Am I missing something? Could this
be a bug?
The fiona version I use is: 1.1.4 (Python 2.7.6 on Ubuntu 14.04 64 bit).
Best regards,
Michael
--
Dr. rer. nat. Michael Nolde
Christian-Albrechts-Universit?t zu Kiel
Geographisches Institut (Sektion Geographie)
Neufeldtstra?e 10, Laurispark - Geb. 32, 24118 Kiel
Tel.: 0431 880-4909
E-Mail: nolde at geographie.uni-kiel.de
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

--
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20140509/45bfabf2/attachment.htm>

Michael Nolde

2014-05-09 22:34:52 UTC

Permalink

Hi Sean,

I uploaded the shapefile together with a demo script here:
http://flatpolar.org/floatval.zip

Executing the script produces the following output with my setup:
Item Nr. 0 : 1.0
Item Nr. 1 : 613.0
Item Nr. 2 : 1.0
Item Nr. 3 : 0.0
Item Nr. 4 : 1.0
Item Nr. 5 : 1.0
Item Nr. 6 : 0.0
Item Nr. 7 : 1.0
Item Nr. 8 : 1.0
Item Nr. 9 : 500.0

while the output should be:
Item Nr. 0 : 1.5
Item Nr. 1 : 613.0
Item Nr. 2 : 1.5
Item Nr. 3 : 0.94
...

The attribute field used is named 'LEISTUNG'.
I'm probably just doing something wrong here, but I'm really stuck.
Thank you very much for your support, this is highly appreciated!

Best regards (and since this is my first post here, hello to everybody).

Michael

Post by Sean Gillies
Hi Michael,
I can't reproduce this error with any data I have. Can you put your
shapefile up for download and tell me which field(s) are the ones being
rounded? I'll take a look if you can.
Yours,
On Fri, May 9, 2014 at 6:14 AM, Michael Nolde

--
Sean Gillies
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

Sean Gillies

2014-05-10 16:03:31 UTC

Permalink

Michael,

I see what you mean. GDAL's ogrinfo reports the same as Fiona:

LEISTUNG (Real) = 1.000000000000000
EINHEIT (String) = MW
BEZUG (String) = FWL
POINT (506130.0 6071821.0)

OGRFeature(test):1
fid (String) = Nolde_Uni_Kiel_Biogas_WKA_Tabelle_Febr_2014_BGA_25832.93
BST_NR (Real) = 54001325650.000000000000000
ANL_NR (String) = 0001
ANLAGENBEZ (String) = Biogasanlage
VIERTE_BIM (String) = 1.2.2.2V
AN_RW (Integer) = 32503337
AN_HW (Integer) = 6073281
LEISTUNG (Real) = 613.000000000000000
EINHEIT (String) = kW
BEZUG (String) = elektr. Leistung
POINT (503337.000000000116415 6073281.0)

OGRFeature(test):2
fid (String) = Nolde_Uni_Kiel_Biogas_WKA_Tabelle_Febr_2014_BGA_25832.94
BST_NR (Real) = 54001325650.000000000000000
ANL_NR (String) = 0001
ANLAGENBEZ (String) = Biogasanlage
VIERTE_BIM (String) = 1.2.2.2V
AN_RW (Integer) = 32503337
AN_HW (Integer) = 6073281
LEISTUNG (Real) = 1.000000000000000
EINHEIT (String) = MW
BEZUG (String) = FWL
POINT (503337.000000000116415 6073281.0)

I opened the test.dbf file in OpenOffice and also saw rounded values.

What software did you use to create the shapefile? With what software are
you seeing the values [1.5, 613.0, 1.5]? Is it possible that precision is
being lost when writing the shapefile?

Post by Michael Nolde
Hi Sean,
http://flatpolar.org/floatval.zip
Item Nr. 0 : 1.0
Item Nr. 1 : 613.0
Item Nr. 2 : 1.0
Item Nr. 3 : 0.0
Item Nr. 4 : 1.0
Item Nr. 5 : 1.0
Item Nr. 6 : 0.0
Item Nr. 7 : 1.0
Item Nr. 8 : 1.0
Item Nr. 9 : 500.0
Item Nr. 0 : 1.5
Item Nr. 1 : 613.0
Item Nr. 2 : 1.5
Item Nr. 3 : 0.94
...
The attribute field used is named 'LEISTUNG'.
I'm probably just doing something wrong here, but I'm really stuck.
Thank you very much for your support, this is highly appreciated!
Best regards (and since this is my first post here, hello to everybody).
Michael

--
Sean Gillies
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

--
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

--
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20140510/c3dd14e8/attachment.htm>

Michael Nolde

2014-05-10 17:08:56 UTC

Permalink

Dear Sean,

this is weird. You're right, I also get rounded values in Open Office,
ogrinfo and in
GRASS. However, when I open the file in QGIS (both version 2.0 and 2.2), I
see decimal places (as is supposedly correct).
See this screenshot: Loading Image...

So the information was not lost during creation of the shapefile.
And funny enough, if I don't use OpenOffice calc but 'Gnumeric', I also see
the correct entries with deciamal places
(Loading Image...

).
This leaves me somewhat confused.

I originally created the file from an XLS-Spreadsheet using the QGIS csv
import tool. My guess now is that the different output originates from
some country specific settings inside the software packages used, maybe
some issue with dot or comma being used as seperator. However, the
original file is displayed correctly in both Open Office and Gnumeric.

Well, whatever it is, it is pretty obvious that the problem does not lie
with fiona. I'm not sure how to proceed. Any suggestions?

Have a nice weekend!
Michael

Post by Sean Gillies
Michael,
LEISTUNG (Real) = 1.000000000000000
EINHEIT (String) = MW
BEZUG (String) = FWL
POINT (506130.0 6071821.0)
OGRFeature(test):1
fid (String) = Nolde_Uni_Kiel_Biogas_WKA_Tabelle_Febr_2014_BGA_25832.93
BST_NR (Real) = 54001325650.000000000000000
ANL_NR (String) = 0001
ANLAGENBEZ (String) = Biogasanlage
VIERTE_BIM (String) = 1.2.2.2V
AN_RW (Integer) = 32503337
AN_HW (Integer) = 6073281
LEISTUNG (Real) = 613.000000000000000
EINHEIT (String) = kW
BEZUG (String) = elektr. Leistung
POINT (503337.000000000116415 6073281.0)
OGRFeature(test):2
fid (String) = Nolde_Uni_Kiel_Biogas_WKA_Tabelle_Febr_2014_BGA_25832.94
BST_NR (Real) = 54001325650.000000000000000
ANL_NR (String) = 0001
ANLAGENBEZ (String) = Biogasanlage
VIERTE_BIM (String) = 1.2.2.2V
AN_RW (Integer) = 32503337
AN_HW (Integer) = 6073281
LEISTUNG (Real) = 1.000000000000000
EINHEIT (String) = MW
BEZUG (String) = FWL
POINT (503337.000000000116415 6073281.0)
I opened the test.dbf file in OpenOffice and also saw rounded values.
What software did you use to create the shapefile? With what software are
you seeing the values [1.5, 613.0, 1.5]? Is it possible that precision is
being lost when writing the shapefile?
On Fri, May 9, 2014 at 4:34 PM, Michael Nolde

Post by Sean Gillies
Hi Michael,
I can't reproduce this error with any data I have. Can you put your
shapefile up for download and tell me which field(s) are the ones

being

Post by Sean Gillies
rounded? I'll take a look if you can.
Yours,
On Fri, May 9, 2014 at 6:14 AM, Michael Nolde

Post by Michael Nolde
Hi there,
I noticed some strange behavior when reading float values from

shapefile