Description

DocumentFactories are the main means by which Documents are ingested into the system. Once the 'load' argument has been called, a DocumentFactory should be able to return, on request, one or more Documents. The way in which it does this will depend on the way in which it has been configured, and how 'load' was called. For example it may locate all documents, and cache them internally (e.g. for multiple XML documents within a single file), or it may crawl, locating and returning the documents one at a time (e.g. for many large files in a directory structure.)

Implementations

• documentFactory.SimpleDocumentFactory
  A 'Smart' DocumentFactory that will attempt to do the most sensible thing to generate documents, based on what is passed to its 'load' method. It does this by calling an appropriate DocumentStream. Classes of DocumentStream implemented at the present time are listed below for reference.
• documentFactory.ComponentDocumentFactory
  A ComponentDocumentFactory is configured ahead of time with an XPath. When its 'load' method is called on a Record, the record is processed, and elements matching the configured XPath are extracted and treated as new documents in their own right.
• documentFactory.AccumulatingDocumentFactory
  [docs coming soon]

Capabilities

At the present time, the following DocumentStreams are available for use by DocumentFactories.
Note Well: these are only intended for use by DocumentFactories, and are unlikely to behave correctly if called directly by users' script.

• documentFactory.FileDocumentStream
  Given the path to a file, find XML documents in it and read them into memory.
• documentFactory.BigFileDocumentStream
  Given the path to a file, find XML documents in it, but only retrieve them on request.
• documentFactory.TarDocumentStream
  Given the data which makes up a tar file, extract the files from it as Documents.
• documentFactory.TarFileDocumentStream
  Given a path to a tar file, extract the files from it as Documents.
• documentFactory.ZipDocumentStream
  Given the data which makes up a zip file, extract the files from it as Documents.
• documentFactory.ZipFileDocumentStream
  Given a path to a zip file, extract the files from it as Documents.
• documentFactory.ClusterFileDocumentStream
  Given the path to a raw cluster data file, merge and create documents.
• documentFactory.MarcFileDocumentStream
  Given the path to a file containing MARC records, extract them.
• documentFactory.BigMarcFileDocumentStream
  Given the path to a file containing MARC records, locate them, but only read into memory as requested.
• documentFactory.Z3950SearchDocumentStream
  Given a tuple of (Zoom Connection, Zoom Query), perform the search and then map the results to Documents.
• documentFactory.SRWSearchDocumentStream
  Given a tuple of (URL, {args}) turn the arguments (ala SRU) into an SRW request, perform it, and map the results to Documents.
• documentFactory.OAIHarvestDocumentStream
  Given a tuple of (URL, {args}) turn the arguments into the parameters for OAI, and harvest the records from the URL, mapping them to Documents.
• documentFactory.GoogleSearchDocumentStream
  Given a Google query, use the Google SOAP interface to search, fetch the actual files matched and map them to documents.
  OBSELETE - Google have withdrawn their SOAP interface.

API

Function	Parameters	Returns	Description
__init__	session, config, parent		The constructer takes the config node for the object, and its parent (usually a database).
load	session, ?data, ?cache, ?format, ?tagName, ?codec		Load the data provide (or use the configured default if not provided). The way the data is loaded is dependent on the other parameters (or their configured defaults if absent): cache - should documents be cached in memory: 0 = No, 1 = Locations cached but not documents, 2 = Yes format - specifies how to treat the data parameter tagName - The XML tag to treat as the document root codec - specifies the codec to use to read the documents in
get_document	session, ?index	Document	Return the index'th document in the factory if index is provided, otherwise return the next document.
register_stream	session, format, class		Register the supplied class of DocumentStream with the document factory for the given format. This class will be used the next time 'load' is called with this format.