ll.url

Module implementing URLs

Home › Python software › ll.urlText · XIST · Python
Python softwarelist of projects

ll.url contains an RFC 2396 compliant implementation of URLs and classes for accessing resource metadata as well as file like classes for reading and writing resource data.

These three levels of functionality are implemented in three classes:

URL

URL objects are the names of resources and can be used and modified, regardless of the fact whether these resources actually exits. URL objects never hits the hard drive or the net.

Connection

Connection objects contain functionality that accesses and changes file metadata (like last modified date, permission bits, directory structure etc.). A connection object can be created by calling the connect method on a URL object.

Resource

Resource objects are file like objects that work with the actual bytes that make up the file data. This functionality lives in the Resource class and its subclasses. Creating a resource is done by calling the open method on a Connection or a URL.

def mime2dt​(s):

def httpdate​(dt):

Return a string suitable for a "Last-Modified" and "Expires" header.

dt is a datetime.datetime object in UTC.

def _urlencode​(query_parts):

class Context​(object):

Calling URL.open or URL.connect Connection object. To avoid constantly creating new connections you can pass a Context object to those methods. Connections will be stored in the Context object and will be reused by those methods.

A Context object can also be used as a context manager (see PEP 346 for more info). This context object will be used for all open and connect calls inside the with block. (Note that after the end of the with block, all connections will be closed.)

def __init__​(self):

def closeall​(self):

Close and drop all connections in this context.

def __enter__​(self):

def __exit__​(self, type, value, traceback):

class ThreadLocalContext​(thread._local):

def getcontext​(context):

class Connection​(object):

A Connection object is used for accessing and modifying the metadata associated with a file. It it created by calling the connect method on a URL object.

def stat​(self, *args, **kwargs):

Return the result of a stat call on the file url.

def lstat​(self, *args, **kwargs):

Return the result of a stat call on the file url. Like stat, but does not follow symbolic links.

def chmod​(self, *args, **kwargs):

Set the access mode of the file url to mode.

def chown​(self, *args, **kwargs):

Change the owner and/or group of the file url.

def lchown​(self, *args, **kwargs):

Change the owner and/or group of the file url (ignoring symbolic links).

def uid​(self, *args, **kwargs):

Return the user id of the owner of the file url.

def gid​(self, *args, **kwargs):

Return the group id the file url belongs to.

def owner​(self, *args, **kwargs):

Return the name of the owner of the file url.

def group​(self, *args, **kwargs):

Return the name of the group the file url belongs to.

def mimetype​(self, url):

Return the mimetype of the file url.

def exists​(self, *args, **kwargs):

Test whether the file url exists.

def isfile​(self, *args, **kwargs):

Test whether the resource url is a file.

def isdir​(self, *args, **kwargs):

Test whether the resource url is a directory.

def ismount​(self, *args, **kwargs):

Test whether the resource url is a mount point.

def access​(self, *args, **kwargs):

Test for access to the file/resource url.

def size​(self, url):

Return the size of the file url.

def imagesize​(self, url):

Return the size of the image url (if the resource is an image file) as a (width, height) tuple. This requires the PIL.

def cdate​(self, url):

Return the "metadate change" date of the file/resource url as a datetime.datetime object in UTC.

def adate​(self, url):

Return the last access date of the file/resource url as a datetime.datetime object in UTC.

def mdate​(self, url):

Return the last modification date of the file/resource url as a datetime.datetime object in UTC.

def resheaders​(self, url):

Return the MIME headers for the file/resource url.

def remove​(self, *args, **kwargs):

Remove the file url.

def rmdir​(self, *args, **kwargs):

Remove the directory url.

def rename​(self, *args, **kwargs):

Renames url to target. This might not work if target has a different scheme than url (or is on a different server).

def chdir​(self, *args, **kwargs):

Change the current directory to url.

def mkdir​(self, *args, **kwargs):

Create the directory url.

def makedirs​(self, *args, **kwargs):

Create the directory url and all intermediate ones.

def listdir​(self, *args, **kwargs):

Return a list of items in the directory url. The elements of the list are URL objects relative to url. With the optional pattern argument, this only lists items whose names match the given pattern.

def files​(self, *args, **kwargs):

Return a list of files in the directory url. The elements of the list are URL objects relative to url. With the optional pattern argument, this only lists items whose names match the given pattern.

def dirs​(self, *args, **kwargs):

Return a list of directories in the directory url. The elements of the list are URL objects relative to url. With the optional pattern argument, this only lists items whose names match the given pattern.

def walk​(self, *args, **kwargs):

Return a recursive iterator over files and subdirectories. The iterator yields URL objects naming each child URL of the directory url and its descendants relative to url. This performs a depth-first traversal, returning each directory before all its children. With the optional pattern argument, only yield items whose names match the given pattern.

def walkfiles​(self, *args, **kwargs):

Return a recursive iterator over files in the directory url. With the optional pattern argument, only yield files whose names match the given pattern.

def walkdirs​(self, *args, **kwargs):

Return a recursive iterator over subdirectories in the directory url. With the optional pattern argument, only yield directories whose names match the given pattern.

def open​(self, *args, **kwargs):

Open url for reading or writing. open returns a Resource object.

Which additional parameters are supported depends on the actual resource created. Some common parameters are:

mode (string)

A string indicating how the file is to be opened (just like the mode argument for the builtin open (e.g. "rb" or "wb").

headers (mapping)

Additional headers to use for an HTTP request.

data (byte string)

Request body to use for an HTTP POST request.

remotepython (string)

Name of the Python interpreter to use on the remote side (used by ssh URLs)

ssh_config (string)

SSH configuration file (used by ssh URLs)

class LocalConnection​(Connection):

def _url2filename​(self, url):

def stat​(self, url):

def lstat​(self, url):

def chmod​(self, url, mode):

def _chown​(self, func, url, owner, group):

def chown​(self, url, owner=None, group=None):

def lchown​(self, url, owner=None, group=None):

def chdir​(self, url):

def mkdir​(self, url, mode=511):

def makedirs​(self, url, mode=511):

def uid​(self, url):

def gid​(self, url):

def owner​(self, url):

def group​(self, url):

def exists​(self, url):

def isfile​(self, url):

def isdir​(self, url):

def ismount​(self, url):

def access​(self, url, mode):

def remove​(self, url):

def rmdir​(self, url):

def rename​(self, url, target):

def listdir​(self, url, pattern=None):

def files​(self, url, pattern=None):

def dirs​(self, url, pattern=None):

def _walk​(self, base, name, pattern, which):

def walk​(self, url, pattern=None):

def walkfiles​(self, url, pattern=None):

def walkdirs​(self, url, pattern=None):

def open​(self, url, mode='rb'):

class SshConnection​(Connection):

def __init__​(self, context, server, remotepython='python', ssh_config=None):

def close​(self):

def _url2filename​(self, url):

def _send​(self, filename, cmd, *args, **kwargs):

def stat​(self, url):

def lstat​(self):

def chmod​(self, url, mode):

def chown​(self, url, owner=None, group=None):

def lchown​(self, url, owner=None, group=None):

def chdir​(self, url):

def mkdir​(self, url, mode=511):

def makedirs​(self, url, mode=511):

def uid​(self, url):

def gid​(self, url):

def owner​(self, url):

def group​(self, url):

def exists​(self, url):

def isfile​(self, url):

def isdir​(self, url):

def ismount​(self, url):

def access​(self, url, mode):

def remove​(self, url):

def rmdir​(self, url):

def _cmdwithtarget​(self, cmdname, url, target):

def rename​(self, url, target):

def listdir​(self, url, pattern=None):

def files​(self, url, pattern=None):

def dirs​(self, url, pattern=None):

def walk​(self, url, pattern=None):

def walkfiles​(self, url, pattern=None):

def walkdirs​(self, url, pattern=None):

def open​(self, url, mode='rb'):

def __repr__​(self):

class URLConnection​(Connection):

def mimetype​(self, url):

def size​(self, url):

def imagesize​(self, url):

def mdate​(self, url):

def resheaders​(self, url):

def open​(self, url, mode='rb', headers=None, data=None):

def here​(scheme='file'):

Return the current directory as an URL object.

def home​(user='', scheme='file'):

Return the home directory of the current user (or the user named user, if user is specified) as an URL object:

>>> url.home()
URL('file:/home/walter/')
>>> url.home("andreas")
URL('file:/home/andreas/')

def root​():

Return a blank root URL, i.e. URL("root:").

def File​(name, scheme='file'):

Turn a filename into an URL object:

>>> url.File("a#b")
URL('file:a%23b')

def Dir​(name, scheme='file'):

Turns a directory name into an URL object, just like File, but ensures that the path is terminated with a /:

>>> url.Dir("a#b")
URL('file:a%23b/')

def Ssh​(user, host, path='~/'):

Return a ssh URL for the user user on the host host with the path path.:var:path (defaulting to the users home directory) must be a path in URL notation (i.e. use / as directory separator):

>>> url.Ssh("root", "www.example.com", "~joe/public_html/index.html")
URL('ssh://root@www.example.com/~joe/public_html/index.html')

If the path starts with ~/ it is relative to this users home directory, if it starts with ~user it's relative to the home directory of the user user. In all othercases the path is considered to be absolute.

def first​(urls):

Return the first URL from urls that exists as a real file or directory. None entries in urls will be skipped.

def firstdir​(urls):

Return the first URL from urls that exists as a real directory. None entries in urls will be skipped.

def firstfile​(urls):

Return the first URL from urls that exists as a real file. None entries in urls will be skipped.

def _import​(filename):

class Resource​(object):

A Resource is a base class that provides a file-like interface to local and remote files, URLs and other resources.

Attributes

Each resource object has the following attributes:

url

The URL for which this resource has been opened (i.e. foo.open().url is foo if foo is a URL object);

name

A string version of url;

closed

A bool specifying whether the resource has been closed (i.e. whether the close method has been called).

Methods

In addition to file methods (like read, readlines, write and close) a resource object might provide the following methods:

finalurl

Return the real URL of the resource (this might be different from the url attribute in case of a redirect).

size

Return the size of the file/resource.

mdate

Return the last modification date of the file/resource as a datetime.datetime object in UTC.

mimetype

Return the mimetype of the file/resource.

imagesize

Return the size of the image (if the resource is an image file) as a (width, height) tuple. This requires the PIL.

def finalurl​(self):

def imagesize​(self):

def __repr__​(self):

class FileResource​(Resource, file):

A subclass of Resource that handles local files.

def __init__​(self, url, mode='rb'):

def size​(self):

def mdate​(self):

def mimetype​(self):

class RemoteFileResource​(Resource):

A subclass of Resource that handles remote files (those using the ssh scheme).

def __init__​(self, connection, url, mode='rb'):

def _send​(self, filename, cmd, *args, **kwargs):

def close​(self):

def read​(self, size=-1):

def readline​(self, size=-1):

def readlines​(self, size=-1):

def __iter__​(self):

def next​(self):

def seek​(self, offset, whence=0):

def tell​(self):

def truncate​(self, size=None):

def write​(self, string):

def writelines​(self, strings):

def flush​(self):

def size​(self):

def mdate​(self):

def mimetype​(self):

class URLResource​(Resource):

A subclass of Resource that handles HTTP, FTP and other URLs (i.e. those that are not handled by FileResource or RemoteFileResource.

def __init__​(self, url, mode='rb', headers=None, data=None):

def __getattr__​(self, name):

def close​(self):

def finalurl​(self):

def mimetype​(self):

def resheaders​(self):

def encoding​(self):

def mdate​(self):

def size​(self):

def read​(self, size=-1):

def readline​(self, size=-1):

def resdata​(self):

def imagesize​(self):

def __iter__​(self):

class SchemeDefinition​(object):

A SchemeDefinition instance defines the properties of a particular URL scheme.

def __init__​(self, scheme, usehierarchy, useserver, usefrag, islocal=False, isremote=False, defaultport=None):

Create a new SchemeDefinition instance. Arguments are:

  • scheme: The name of the scheme;

  • usehierarchy: Specifies whether this scheme uses hierarchical URLs or opaque URLs (i.e. whether hier_part or opaque_part from the BNF in RFC 2396 is used);

  • useserver: Specifies whether this scheme uses an Internet-based server authority component or a registry of naming authorities (only for hierarchical URLs);

  • usefrag: Specifies whether this scheme uses fragments (according to the BNF in RFC 2396 every scheme does, but it doesn't make sense for e.g. "javascript", "mailto" or "tel");

  • islocal: Specifies whether URLs with this scheme refer to local files;

  • isremote: Specifies whether URLs with this scheme refer to remote files (there may be schemes which are neither local nor remote, e.g. "mailto");

  • defaultport: The default port for this scheme (only for schemes using server based authority).

def connect​(self, url, context=None, **kwargs):

Create a Connection for the URL url (which must have self as the scheme).

def _connect​(self, url, context=None, **kwargs):

def open​(self, *args, **kwargs):

def closeall​(self, context):

Close all connections active for this scheme in the context context.

def __repr__​(self):

class LocalSchemeDefinition​(SchemeDefinition):

def open​(self, *args, **kwargs):

class SshSchemeDefinition​(SchemeDefinition):

def _connect​(self, url, context=None, **kwargs):

def open​(self, url, mode='rb', context=None, remotepython='python', ssh_config=None):

def closeall​(self, context):

class Path​(object):

def __init__​(self, path=None):

def _fixsegment​(cls, segment):

def _prefix​(cls, path):

def insert​(self, index, *others):

def startswith​(self, prefix):

Return whether self starts with the path prefix. prefix will be converted to a Path if it isn't one.

def endswith​(self, suffix):

Return whether self ends with the path suffix. suffix will be converted to a Path if it isn't one. If suffix is absolute a normal comparison will be done.

def clone​(self):

def __repr__​(self):

def __str__​(self):

def __eq__​(self, other):

def __ne__​(self, other):

def __hash__​(self):

def __len__​(self):

def __getitem__​(self, index):

def __setitem__​(self, index, value):

def __delitem__​(self, index):

def __contains__​(self, item):

def __getslice__​(self, index1, index2):

Return of slice of the path. The resulting path will always be relative, i.e. the leading / will be dropped.

def __setslice__​(self, index1, index2, seq):

def __delslice__​(self, index1, index2):

property isabs:

Is the path absolute?

def __get__​(self):

def __set__​(self, isabs):

def __delete__​(self):

def _segments2path​(cls, segments):

def _path2segments​(cls, path):

def _setpathorsegments​(self, path):

property path:

The complete path as a string.

def __get__​(self):

def __set__​(self, path):

def __delete__​(self):

property segments:

The path as a list of (name, param) tuples.

def __get__​(self):

def __set__​(self, path):

def __delete__​(self):

property file:

The filename without the path, i.e. the name part of the last component of path. The baz.html part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, file):

Setting the filename preserves the parameter in the last segment.

def __delete__​(self):

Deleting the filename preserves the parameter in the last segment.

property ext:

The filename extension of the last segment of the path. The html part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, ext):

Setting the extension preserves the parameter in the last segment.

def __delete__​(self):

Deleting the extension preserves the parameter in the last segment.

def withext​(self, ext):

Return a new Path where the filename extension has been replaced with ext.

def withoutext​(self):

Return a new Path where the filename extension has been removed.

def withfile​(self, file):

Return a new Path where the filename (i.e. the name of the last component of segments) has been replaced with file.

def withoutfile​(self):

Return a new Path where the filename (i.e. the name of the last component of segments) has been removed.

def clear​(self):

def __div__​(self, other):

Join two paths.

def __rdiv__​(self, other):

Right hand version of __div__. This supports list and generators as the left hand side too.

def relative​(self, basepath):

Return an relative Path rel such that basepath/rel == self`, i.e. this is the inverse operation of __div__.

If self is relative, an identical copy of self will be returned.

def reverse​(self):

def normalize​(self):

def normalized​(self):

def local​(self):

Return self converted to a filename using the file naming conventions of the OS. Parameters will be dropped in the resulting string.

def abs​(self):

Return an absolute version of self.

def real​(self):

Return the canonical version of self, eliminating all symbolic links.

class Query​(dict):

def __init__​(self, arg=None, **kwargs):

def __setitem__​(self, key, value):

def add​(self, key, *values):

def __xrepr__​(self, mode='default'):

class URL​(object):

An RFC 2396 compliant URL.

def __init__​(self, url=None):

Create a new URL instance. url may be a str or unicode instance, or an URL (in which case you'll get a copy of url), or None (which will create an URL referring to the "current document").

def _clear​(self):

def clone​(self):

Return an identical copy self.

property scheme:

The URL scheme (e.g. ftp, ssh, http or mailto). The scheme will be None if the URL is a relative one.

def __get__​(self):

def __set__​(self, scheme):

The scheme will be converted to lowercase on setting (if scheme is not None, otherwise the scheme will be deleted).

def __delete__​(self):

Deletes the scheme, i.e. makes the URL relative.

property userinfo:

The user info part of the URL; i.e. the user part of http://user@www.example.com:8080/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, userinfo):

def __delete__​(self):

property host:

The host part of the URL; i.e. the www.example.com part of http://user@www.example.com:8080/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, host):

def __delete__​(self):

property port:

The port number of the URL (as an int) or None if the URL has none. The 8080 in http://user@www.example.com:8080/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, port):

def __delete__​(self):

property hostport:

The host and (if specified) the port number of the URL, i.e. the www.example.com:8080 in http://user@www.example.com:8080/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, hostport):

def __delete__​(self):

property server:

The server part of the URL; i.e. the user@www.example.com part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, server):

Setting the server always works even if the current scheme does use opaque_part or reg_name but will be ignored when reassembling the URL for the url property.

def __delete__​(self):

property reg_name:

The reg_name part of the URL for hierarchical schemes that use a name based authority instead of server.

def __get__​(self):

def __set__​(self, reg_name):

def __delete__​(self):

property authority:

The authority part of the URL for hierarchical schemes. Depending on the scheme, this is either server or reg_name.

def __get__​(self):

def __set__​(self, authority):

def __delete__​(self):

property isabspath:

Specifies whether the path of a hierarchical URL is absolute, (i.e. it has a leading "/"). Note that the path will always be absolute if an authority is specified.

def __get__​(self):

def __set__​(self, isabspath):

property path:

The path segments of a hierarchical URL as a Path object.

def __get__​(self):

def __set__​(self, path):

def __delete__​(self):

property file:

The filename without the path, i.e. the name part of the last component of path. The baz.html part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, file):

Setting the filename preserves the parameter in the last segment.

def __delete__​(self):

Deleting the filename preserves the parameter in the last segment.

property ext:

The filename extension of the last segment of the path. The html part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, ext):

Setting the extension preserves the parameter in the last segment.

def __delete__​(self):

Deleting the extension preserves the parameter in the last segment.

property query_parts:

The query component as a dictionary, i.e. {u"spam": u"eggs"} from http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

If the query component couldn't be parsed, query_parts will be False.

def __get__​(self):

def __set__​(self, query_parts):

def __delete__​(self):

property query:

The query component, i.e. the spam=eggs part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, query):

def __delete__​(self):

property opaque_part:

The opaque part (for schemes like mailto that are not hierarchical).

def __get__​(self):

def __set__​(self, opaque_part):

def __delete__​(self):

property frag:

The fragment identifier, which references a part of the resource, i.e. the frag part of http://user@www.example.com/bar/baz.html;xyzzy?spam=eggs#frag.

def __get__​(self):

def __set__​(self, frag):

def __delete__​(self):

property url:

The complete URL

def __get__​(self):

Getting url reassembles the URL from the components.

def __set__​(self, url):

Setting url parses url into the components. url may also be an URL instance, in which case the URL will be copied.

def __delete__​(self):

After deleting the URL the resulting object will refer to the "current document".

def withext​(self, ext):

Return a new URL where the filename extension has been replaced with ext.

def withoutext​(self):

Return a new URL where the filename extension has been removed.

def withfile​(self, file):

Return a new URL where the filename (i.e. the name of last component of path_segments) has been replaced with file.

def withoutfile​(self):

def withfrag​(self, frag):

Return a new URL where the fragment has been replaced with frag.

def withoutfrag​(self):

Return a new URL where the frag has been dropped.

def __div__​(self, other):

Join self with another (possible relative) URL other, to form a new URL.

other may be a str, unicode or URL object. It may be None (referring to the "current document") in which case self will be returned. It may also be a list or other iterable. For this case a list (or iterator) will be returned where __div__ will be applied to every item in the list/iterator. E.g. the following expression returns all the files in the current directory as absolute URLs (see the method files and the function here for further explanations):

>>> here = url.here()
>>> for f in here/here.files():
...     print f

def __rdiv__​(self, other):

Right hand version of __div__. This supports lists and iterables as the left hand side too.

def relative​(self, baseurl):

Return an relative URL rel such that baseurl/rel == self, i.e. this is the inverse operation of __div__.

If self is relative, has a different scheme or authority than baseurl or a non-hierarchical scheme, an identical copy of self will be returned.

def __str__​(self):

def __unicode__​(self):

def __repr__​(self):

def __nonzero__​(self):

Return whether the URL is not empty, i.e. whether it is not the URL referring to the start of the current document.

def __eq__​(self, other):

Return whether two URL objects are equal. Note that only properties relevant for the current scheme will be compared.

def __ne__​(self, other):

Return whether two URL objects are not equal.

def __hash__​(self):

Return a hash value for self, to be able to use URL objects as dictionary keys. You must be careful not to modify an URL as soon as you use it as a dictionary key.

def abs​(self, scheme=-1):

Return an absolute version of self (works only for local URLs).

If the argument scheme is specified, it will be used for the resulting URL otherwise the result will have the same scheme as self.

def real​(self, scheme=-1):

Return the canonical version of self, eliminating all symbolic links (works only for local URLs).

If the argument scheme is specified, it will be used for the resulting URL otherwise the result will have the same scheme as self.

def islocal​(self):

Return whether self refers to a local file, i.e. whether self is a relative URL or the scheme is root or file).

def _checklocal​(self):

def local​(self):

Return self as a local filename (which will only works if self is local (see islocal).

def _connect​(self, context=None, **kwargs):

def connect​(self, context=None, **kwargs):

Return a Connection object for accessing and modifying the metadata of self.

Whether you get a new connection object, or an existing one depends on the scheme, the URL itself, and the context passed in (as the context argument).

def open​(self, mode='rb', context=None, *args, **kwargs):

Open self for reading or writing. open returns a Resource object.

Which additional parameters are supported depends on the actual resource created. Some common parameters are:

mode (supported by all resources)

A string indicating how the file is to be opened (just like the mode argument for the builtin open; e.g. "rb" or "wb").

context (supported by all resources)

open needs a Connection for this URL which it gets from a Context object.

headers

Additional headers to use for an HTTP request.

data

Request body to use for an HTTP POST request.

remotepython

Name of the Python interpreter to use on the remote side (used by ssh URLs)

ssh_config

SSH configuration file (used by ssh URLs)

def openread​(self, context=None, *args, **kwargs):

def openwrite​(self, context=None, *args, **kwargs):

def __getattr__​(self, name):

__getattr__ forwards every unresolved attribute access to the appropriate connection. This makes it possible to call Connection methods directly on URL objects:

>>> from ll import url
>>> u = url.URL("file:README")
>>> u.size()
1584L

instead of:

>>> from ll import url
>>> u = url.URL("file:README")
>>> u.connect().size(u)
1584L

def import_​(self, mode='always'):

import the file as a Python module. The file extension will be ignored, which means that you might not get exactly the file you specified. mode can have the following values:

"always" (the default)

The module will be imported on every call;

"once"

The module will be imported only on the first call;

"new"

The module will be imported every time it has changed since the last call.

def __iter__​(self):

def __xrepr__​(self, mode='default'):