Frontera API¶
This section documents the Frontera core API, and is intended for developers of middlewares and backends.
Frontera API / Manager¶
The main entry point to Frontera API is the FrontierManager
object, passed to middlewares and backend through the from_manager class method. This object provides access to all
Frontera core components, and is the only way for middlewares and backend to access them and hook their
functionality into Frontera.
The FrontierManager
is responsible for loading the installed
middlewares and backend, as well as for managing the data flow around the whole frontier.
Loading from settings¶
Although FrontierManager
can be initialized using parameters the most
common way of doing this is using Frontera Settings.
This can be done through the from_settings
class method, using either a string path:
>>> from frontera import FrontierManager
>>> frontier = FrontierManager.from_settings('my_project.frontier.settings')
or a BaseSettings
object instance:
>>> from frontera import FrontierManager, Settings
>>> settings = Settings()
>>> settings.MAX_PAGES = 0
>>> frontier = FrontierManager.from_settings(settings)
It can also be initialized without parameters, in this case the frontier will use the default settings:
>>> from frontera import FrontierManager, Settings
>>> frontier = FrontierManager.from_settings()
Frontier Manager¶
Starting/Stopping the frontier¶
Sometimes, frontier components need to perform initialization and finalization operations. The frontier mechanism to
notify the different components of the frontier start and stop is done by the
start()
and
stop()
methods respectively.
By default auto_start
frontier value is activated,
this means that components will be notified once the
FrontierManager
object is created.
If you need to have more fine control of when different components are initialized, deactivate
auto_start
and manually call frontier API
start()
and
stop()
methods.
Note
Frontier stop()
method is not automatically called
when auto_start
is active (because frontier is
not aware of the crawling state). If you need to notify components of frontier end you should call the method
manually.
Frontier iterations¶
Once frontier is running, the usual process is the one described in the data flow section.
Crawler asks the frontier for next pages using the
get_next_requests()
method.
Each time the frontier returns a non empty list of pages (data available), is what we call a frontier iteration.
Current frontier iteration can be accessed using the
iteration
attribute.
Finishing the frontier¶
Crawl can be finished either by the Crawler or by the Frontera. Frontera will finish when a maximum number
of pages is returned. This limit is controlled by the
max_requests
attribute
(MAX_REQUESTS
setting).
If max_requests
has a value of 0 (default value)
the frontier will continue indefinitely.
Once the frontier is finished, no more pages will be returned by the
get_next_requests
method and
finished
attribute will be True.
Component objects¶
-
class
frontera.core.components.
Component
¶ Interface definition for a frontier component The
Component
object is the base class for frontierMiddleware
andBackend
objects.FrontierManager
communicates with the active components using the hook methods listed below.Implementations are different for
Middleware
andBackend
objects, therefore methods are not fully described here but in their corresponding section.Attributes
-
name
¶ The component name
Abstract methods
-
frontier_start
()¶ Called when the frontier starts, see starting/stopping the frontier.
-
frontier_stop
()¶ Called when the frontier stops, see starting/stopping the frontier.
-
page_crawled
(response)¶ This method is called every time a page has been crawled.
Parameters: response (object) – The Response
object for the crawled page.
-
request_error
(page, error)¶ This method is called each time an error occurs when crawling a page.
Parameters: - request (object) – The crawled with error
Request
object. - error (string) – A string identifier for the error.
- request (object) – The crawled with error
Class Methods
-
classmethod
from_manager
(manager)¶ Class method called from
FrontierManager
passing the manager itself.Example of usage:
def from_manager(cls, manager): return cls(settings=manager.settings)
-
Test mode¶
In some cases while testing, frontier components need to act in a different way than they usually do (for instance
domain middleware accepts non valid URLs like 'A1'
or 'B1'
when parsing
domain urls in test mode).
Components can know if the frontier is in test mode via the boolean
test_mode
attribute.
Other ways of using the frontier¶
Communication with the frontier can also be done through other mechanisms such as an HTTP API or a queue system. These functionalities are not available for the time being, but hopefully will be included in future versions.