[Community] cookies and OWSlib

[David Winslow]

Hi all,

I work on GeoNode (http://geonode.org/). We're using OWSLib to fetch data
from GeoNetwork. Recently I've discovered that due to the way GeoNetwork
handles sessions, it makes a big difference for us to include HTTP cookies
on our service requests (without them, GeoNetwork creates a new session for
each service request and eventually locks up after running out of RAM.)

Therefore I would like to modify owslib to use a (persistent) instance of
urllib2.HTTPCookieProcessor when opening URLs, one which I can get a
reference to so that I can log in to GeoNetwork and have owslib use the
session cookie. Ideally, there would be a way to provide my own instance of
urllib2.OpenerDirector which I can customize as needed.

I am currently using a monkey patch to fix this behavior, but it's not the
way I'd like to do things (you can see the diff in GeoNode here:
https://github.com/dwins/geonode/compare/master...csw-cookies ).

For a deeper fix I see basically 3 options:

1) Use Python's urllib2.install_opener to set a default opener instance to
use, and be careful not to pass arguments into owslib methods that would
cause them to create their own openers (for example, including a username
and password in a call to owslib.util.openURL).
2) Create a module-level variable (owslib.util.opener) and modify
owslib.util.openURL and owslib.util.http_post to reference it exclusively.
Then client code which needs to modify the HTTP handling behavior can
simply replace this variable.
3) Make each client class (ie, owslib.csw.CatalogueServiceWeb) use its own
opener instance which may be provided by client code.

The attached patch is a partial implementation of approach #3. (In each of
these cases there is the question of how to handle the username/password
arguments to owslib.util.openURL, I haven't addressed this.)

Is this sort of thing something the OWSLib community would be interested in
incorporating? If so I'm happy to update this patch to incorporate feedback
etc.

--
David Winslow
OpenGeo - http://opengeo.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20110425/fc9711da/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: owslib-cookies.patch
Type: text/x-patch
Size: 15597 bytes
Desc: not available
URL: <http://lists.gispython.org/pipermail/community/attachments/20110425/fc9711da/attachment.bin>

[Dominic Lowe]

Hi David,

I'd be very happy to see that patch - I'm sure there are other
circumstances when cookies would be useful in OWSLib.

Am I right in thinking your current patch does not support http
password/username logon? I think that's pretty important to maintain as
that was a requirement from some other users.

Otherwise I'm pretty happy for you to suggest a patch. Tom Kralidis may
have more comments as he has been most active on OWSLib recently.

Regards,
Dominic

Post by David Winslow
Hi all,
I work on GeoNode (http://geonode.org/). We're using OWSLib to fetch data from GeoNetwork. Recently I've discovered that due to the way GeoNetwork handles sessions, it makes a big difference for us to include HTTP cookies on our service requests (without them, GeoNetwork creates a new session for each service request and eventually locks up after running out of RAM.)
Therefore I would like to modify owslib to use a (persistent) instance of urllib2.HTTPCookieProcessor when opening URLs, one which I can get a reference to so that I can log in to GeoNetwork and have owslib use the session cookie. Ideally, there would be a way to provide my own instance of urllib2.OpenerDirector which I can customize as needed.
I am currently using a monkey patch to fix this behavior, but it's not the way I'd like to do things (you can see the diff in GeoNode here: https://github.com/dwins/geonode/compare/master...csw-cookies ).
1) Use Python's urllib2.install_opener to set a default opener instance to use, and be careful not to pass arguments into owslib methods that would cause them to create their own openers (for example, including a username and password in a call to owslib.util.openURL).
2) Create a module-level variable (owslib.util.opener) and modify owslib.util.openURL and owslib.util.http_post to reference it exclusively. Then client code which needs to modify the HTTP handling behavior can simply replace this variable.
3) Make each client class (ie, owslib.csw.CatalogueServiceWeb) use its own opener instance which may be provided by client code.
The attached patch is a partial implementation of approach #3. (In each of these cases there is the question of how to handle the username/password arguments to owslib.util.openURL, I haven't addressed this.)
Is this sort of thing something the OWSLib community would be interested in incorporating? If so I'm happy to update this patch to incorporate feedback etc.
--
David Winslow
OpenGeo - http://opengeo.org/

[Kralidis,Tom [Ontario]]

Hi David: thanks for the info. As discussed on irc, I think this is
great addition (as an optional arg). +1 for approach #3. Please open a
ticket (with patch, when ready) at
http://trac.gispython.org/lab/wiki/OwsLib and we will make sure this
gets implemented.

Thanks

..Tom

-----Original Message-----
From: community-bounces at lists.gispython.org
[mailto:community-bounces at lists.gispython.org] On Behalf Of
Dominic Lowe
Sent: Tuesday, 26 April 2011 10:46
To: community at lists.gispython.org
Subject: Re: [Community] cookies and OWSlib
Hi David,
I'd be very happy to see that patch - I'm sure there are
other circumstances when cookies would be useful in OWSLib.
Am I right in thinking your current patch does not support
http password/username logon? I think that's pretty important
to maintain as that was a requirement from some other users.
Otherwise I'm pretty happy for you to suggest a patch. Tom
Kralidis may have more comments as he has been most active on
OWSLib recently.
Regards,
Dominic

Post by David Winslow
Hi all,
I work on GeoNode (http://geonode.org/). We're using

OWSLib to fetch

Post by David Winslow
data from GeoNetwork. Recently I've discovered that due to the way
GeoNetwork handles sessions, it makes a big difference for us to
include HTTP cookies on our service requests (without them,

GeoNetwork

Post by David Winslow
creates a new session for each service request and

eventually locks up

Post by David Winslow
after running out of RAM.)
Therefore I would like to modify owslib to use a

(persistent) instance of urllib2.HTTPCookieProcessor when
opening URLs, one which I can get a reference to so that I
can log in to GeoNetwork and have owslib use the session
cookie. Ideally, there would be a way to provide my own
instance of urllib2.OpenerDirector which I can customize as needed.

Post by David Winslow
I am currently using a monkey patch to fix this behavior,

but it's not the way I'd like to do things (you can see the
https://github.com/dwins/geonode/compare/master...csw-cookies ).

Post by David Winslow
1) Use Python's urllib2.install_opener to set a default

opener instance to use, and be careful not to pass arguments
into owslib methods that would cause them to create their own
openers (for example, including a username and password in a
call to owslib.util.openURL).

Post by David Winslow
2) Create a module-level variable (owslib.util.opener) and

modify owslib.util.openURL and owslib.util.http_post to
reference it exclusively. Then client code which needs to
modify the HTTP handling behavior can simply replace this variable.

Post by David Winslow
3) Make each client class (ie,

owslib.csw.CatalogueServiceWeb) use its own opener instance
which may be provided by client code.

Post by David Winslow
The attached patch is a partial implementation of approach #3. (In
each of these cases there is the question of how to handle the
username/password arguments to owslib.util.openURL, I haven't
addressed this.)
Is this sort of thing something the OWSLib community would

be interested in incorporating? If so I'm happy to update
this patch to incorporate feedback etc.

Post by David Winslow
--
David Winslow
OpenGeo - http://opengeo.org/

--
Scanned by iCritical.
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

[David Winslow]

The patch I have does not modify the existing password support - so when you
pass in username/password to the openURL function, a new OpenerDirector is
created, configured and used for that one request and immediately thrown
away, ignoring the new "opener" field I have added to the various OWS client
classes. I'll try and unify these for the final patch, now that I've gotten
some feedback on the approach I've chosen.

--
David Winslow
OpenGeo - http://opengeo.org/

Post by Dominic Lowe
Hi David,
I'd be very happy to see that patch - I'm sure there are other
circumstances when cookies would be useful in OWSLib.
Am I right in thinking your current patch does not support http
password/username logon? I think that's pretty important to maintain as that
was a requirement from some other users.
Otherwise I'm pretty happy for you to suggest a patch. Tom Kralidis may
have more comments as he has been most active on OWSLib recently.
Regards,
Dominic

Post by David Winslow
Hi all,
I work on GeoNode (http://geonode.org/). We're using OWSLib to fetch
data from GeoNetwork. Recently I've discovered that due to the way
GeoNetwork handles sessions, it makes a big difference for us to include
HTTP cookies on our service requests (without them, GeoNetwork creates a new
session for each service request and eventually locks up after running out
of RAM.)
Therefore I would like to modify owslib to use a (persistent) instance of
urllib2.HTTPCookieProcessor when opening URLs, one which I can get a
reference to so that I can log in to GeoNetwork and have owslib use the
session cookie. Ideally, there would be a way to provide my own instance of
urllib2.OpenerDirector which I can customize as needed.
I am currently using a monkey patch to fix this behavior, but it's not the
https://github.com/dwins/geonode/compare/master...csw-cookies ).
1) Use Python's urllib2.install_opener to set a default opener instance to
use, and be careful not to pass arguments into owslib methods that would
cause them to create their own openers (for example, including a username
and password in a call to owslib.util.openURL).
2) Create a module-level variable (owslib.util.opener) and modify
owslib.util.openURL and owslib.util.http_post to reference it exclusively.
Then client code which needs to modify the HTTP handling behavior can
simply replace this variable.
3) Make each client class (ie, owslib.csw.CatalogueServiceWeb) use its own
opener instance which may be provided by client code.
The attached patch is a partial implementation of approach #3. (In each of
these cases there is the question of how to handle the username/password
arguments to owslib.util.openURL, I haven't addressed this.)
Is this sort of thing something the OWSLib community would be interested
in incorporating? If so I'm happy to update this patch to incorporate
feedback etc.
--
David Winslow
OpenGeo - http://opengeo.org/

--
Scanned by iCritical.
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20110427/5a0f6012/attachment.htm>

[David Winslow]

Ok, I have prepared a more thorough implementation of this idea. Although
some unit tests were failing for me (looks like some servers have been
updated/reconfigured since the tests were last updated) I have verified that
this patch doesn't break any tests that were already passing, and that it
works properly in GeoNode (ie no more session proliferation).

I don't believe I have an account on the OWSLib trac and self-registration
does not seem to be enabled, but the patch is attached to this email (I can
make it available via HTTP as well if that would be more useful; it wasn't
clear if my last attachment to this list made it through or not.)

There's a lot changed internally (everything is standardized on going
through local "opener" variables now) but for the factory methods used there
are just a few new optional arguments when setting up the client object
(these should work for WebFeatureService, WebMapService, WebCoverageService,
and CatalogueServiceWeb):

*opener* = a urllib2.OpenerDirector. if the other parameters listed here
are provided then they will be added to the opener.

*cookies* = a cookielib.CookieJar implementation (compatibility note! where
cookies were allowed before they were expected as a literal string, I have
used a CookieJar as this is the only option to work with the builtin
urllib2.HTTPCookieProcessor. However, when I tried to test the previous
cookie code I couldn't get it to work due to an apparent bug in the code so
I would be surprised if client code is relying on that functionality. I can
elaborate more on the bug I think I found but that code is replaced in the
patch.)

*username* = a username for http basic authentication (only used if a
password is also provided)
*password* = a password for http basic authentication (only used if a
username is also provided)

--
David Winslow
OpenGeo - http://opengeo.org/

Post by David Winslow
The patch I have does not modify the existing password support - so when
you pass in username/password to the openURL function, a new OpenerDirector
is created, configured and used for that one request and immediately thrown
away, ignoring the new "opener" field I have added to the various OWS client
classes. I'll try and unify these for the final patch, now that I've gotten
some feedback on the approach I've chosen.
--
David Winslow
OpenGeo - http://opengeo.org/

Post by Dominic Lowe
Hi David,
I'd be very happy to see that patch - I'm sure there are other
circumstances when cookies would be useful in OWSLib.
Am I right in thinking your current patch does not support http
password/username logon? I think that's pretty important to maintain as that
was a requirement from some other users.
Otherwise I'm pretty happy for you to suggest a patch. Tom Kralidis may
have more comments as he has been most active on OWSLib recently.
Regards,
Dominic

Post by David Winslow
Hi all,
I work on GeoNode (http://geonode.org/). We're using OWSLib to fetch
data from GeoNetwork. Recently I've discovered that due to the way
GeoNetwork handles sessions, it makes a big difference for us to include
HTTP cookies on our service requests (without them, GeoNetwork creates a new
session for each service request and eventually locks up after running out
of RAM.)
Therefore I would like to modify owslib to use a (persistent) instance of
urllib2.HTTPCookieProcessor when opening URLs, one which I can get a
reference to so that I can log in to GeoNetwork and have owslib use the
session cookie. Ideally, there would be a way to provide my own instance of
urllib2.OpenerDirector which I can customize as needed.
I am currently using a monkey patch to fix this behavior, but it's not
https://github.com/dwins/geonode/compare/master...csw-cookies ).
1) Use Python's urllib2.install_opener to set a default opener instance
to use, and be careful not to pass arguments into owslib methods that would
cause them to create their own openers (for example, including a username
and password in a call to owslib.util.openURL).
2) Create a module-level variable (owslib.util.opener) and modify
owslib.util.openURL and owslib.util.http_post to reference it exclusively.
Then client code which needs to modify the HTTP handling behavior can
simply replace this variable.
3) Make each client class (ie, owslib.csw.CatalogueServiceWeb) use its
own opener instance which may be provided by client code.
The attached patch is a partial implementation of approach #3. (In each
of these cases there is the question of how to handle the username/password
arguments to owslib.util.openURL, I haven't addressed this.)
Is this sort of thing something the OWSLib community would be interested
in incorporating? If so I'm happy to update this patch to incorporate
feedback etc.
--
David Winslow
OpenGeo - http://opengeo.org/

--
Scanned by iCritical.
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gispython.org/pipermail/community/attachments/20110427/4ccec07a/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: owslib-cookies-full.patch
Type: text/x-patch
Size: 31421 bytes
Desc: not available
URL: <http://lists.gispython.org/pipermail/community/attachments/20110427/4ccec07a/attachment-0001.bin>

[Kralidis,Tom [Ontario]]

Thanks David. FYI this has been added to trac at
http://trac.gispython.org/lab/ticket/259.

..Tom

-----Original Message-----
From: community-bounces at lists.gispython.org
[mailto:community-bounces at lists.gispython.org] On Behalf Of
David Winslow
Sent: Wednesday, 27 April 2011 18:50
To: gispython.org community projects
Subject: Re: [Community] cookies and OWSlib
Ok, I have prepared a more thorough implementation of this
idea. Although some unit tests were failing for me (looks
like some servers have been updated/reconfigured since the
tests were last updated) I have verified that this patch
doesn't break any tests that were already passing, and that
it works properly in GeoNode (ie no more session proliferation).
I don't believe I have an account on the OWSLib trac and
self-registration does not seem to be enabled, but the patch
is attached to this email (I can make it available via HTTP
as well if that would be more useful; it wasn't clear if my
last attachment to this list made it through or not.)
There's a lot changed internally (everything is standardized
on going through local "opener" variables now) but for the
factory methods used there are just a few new optional
arguments when setting up the client object (these should
work for WebFeatureService, WebMapService,
opener = a urllib2.OpenerDirector. if the other parameters
listed here are provided then they will be added to the opener.
cookies = a cookielib.CookieJar implementation (compatibility
note! where cookies were allowed before they were expected as
a literal string, I have used a CookieJar as this is the only
option to work with the builtin urllib2.HTTPCookieProcessor.
However, when I tried to test the previous cookie code I
couldn't get it to work due to an apparent bug in the code so
I would be surprised if client code is relying on that
functionality. I can elaborate more on the bug I think I
found but that code is replaced in the patch.)
username = a username for http basic authentication (only
used if a password is also provided)
password = a password for http basic authentication (only
used if a username is also provided)
--
David Winslow
OpenGeo - http://opengeo.org/
On Wed, Apr 27, 2011 at 10:15 AM, David Winslow
The patch I have does not modify the existing password
support - so when you pass in username/password to the
openURL function, a new OpenerDirector is created, configured
and used for that one request and immediately thrown away,
ignoring the new "opener" field I have added to the various
OWS client classes. I'll try and unify these for the final
patch, now that I've gotten some feedback on the approach I've chosen.
--
David Winslow
OpenGeo - http://opengeo.org/
On Tue, Apr 26, 2011 at 10:45 AM, Dominic Lowe
Hi David,
I'd be very happy to see that patch - I'm sure
there are other circumstances when cookies would be useful in OWSLib.
Am I right in thinking your current patch does
not support http password/username logon? I think that's
pretty important to maintain as that was a requirement from
some other users.
Otherwise I'm pretty happy for you to suggest a
patch. Tom Kralidis may have more comments as he has been
most active on OWSLib recently.
Regards,
Dominic
Hi all,
I work on GeoNode
(http://geonode.org/). We're using OWSLib to fetch data from
GeoNetwork. Recently I've discovered that due to the way
GeoNetwork handles sessions, it makes a big difference for us
to include HTTP cookies on our service requests (without
them, GeoNetwork creates a new session for each service
request and eventually locks up after running out of RAM.)
Therefore I would like to modify owslib
to use a (persistent) instance of urllib2.HTTPCookieProcessor
when opening URLs, one which I can get a reference to so that
I can log in to GeoNetwork and have owslib use the session
cookie. Ideally, there would be a way to provide my own
instance of urllib2.OpenerDirector which I can customize as needed.
I am currently using a monkey patch to
fix this behavior, but it's not the way I'd like to do things
https://github.com/dwins/geonode/compare/master...csw-cookies ).
1) Use Python's urllib2.install_opener
to set a default opener instance to use, and be careful not
to pass arguments into owslib methods that would cause them
to create their own openers (for example, including a
username and password in a call to owslib.util.openURL).
2) Create a module-level variable
(owslib.util.opener) and modify owslib.util.openURL and
owslib.util.http_post to reference it exclusively. Then
client code which needs to modify the HTTP handling behavior
can simply replace this variable.
3) Make each client class (ie,
owslib.csw.CatalogueServiceWeb) use its own opener instance
which may be provided by client code.
The attached patch is a partial
implementation of approach #3. (In each of these cases there
is the question of how to handle the username/password
arguments to owslib.util.openURL, I haven't addressed this.)
Is this sort of thing something the
OWSLib community would be interested in incorporating? If so
I'm happy to update this patch to incorporate feedback etc.
--
David Winslow
OpenGeo - http://opengeo.org/
--
Scanned by iCritical.
_______________________________________________
Community mailing list
Community at lists.gispython.org
http://lists.gispython.org/mailman/listinfo/community

[Sean Gillies]

Post by David Winslow
Hi all,
I work on GeoNode (http://geonode.org/). ?We're using OWSLib to fetch data
from GeoNetwork. ?Recently I've discovered that due to the way GeoNetwork
handles sessions, it makes a big difference for us to include HTTP cookies
on our service requests (without them, GeoNetwork creates a new session for
each service request and eventually locks up after running out of RAM.)
Therefore I would like to modify owslib to use a (persistent) instance of
urllib2.HTTPCookieProcessor when opening URLs, one which I can get a
reference to so that I can log in to GeoNetwork and have owslib use the
session cookie. ?Ideally, there would be a way to provide my own instance of
urllib2.OpenerDirector which I can customize as needed.
I am currently using a monkey patch to fix this behavior, but it's not the
way I'd like to do things (you can see the diff in GeoNode
here:?https://github.com/dwins/geonode/compare/master...csw-cookies ).
1) Use Python's urllib2.install_opener to set a default opener instance to
use, and be careful not to pass arguments into owslib methods that would
cause them to create their own openers (for example, including a username
and password in a call to owslib.util.openURL).
2) Create a module-level variable (owslib.util.opener) and modify
owslib.util.openURL and owslib.util.http_post to reference it exclusively.
?Then client code which needs to modify the HTTP handling behavior can
simply replace this variable.
3) Make each client class (ie, owslib.csw.CatalogueServiceWeb) use its own
opener instance which may be provided by client code.
The attached patch is a partial implementation of approach #3. (In each of
these cases there is the question of how to handle the username/password
arguments to owslib.util.openURL, I haven't addressed this.)
Is this sort of thing something the OWSLib community would be interested in
incorporating? ?If so I'm happy to update this patch to incorporate feedback
etc.

Hi David,

#3 sounds right to me, I've seen this optional callable parameter
pattern in other APIs - simplejson's object_hook, for example.