pqueue_fetcher Implements a priority-queue-based fetching system
Download

pqueue_fetcher Ranking & Summary

pqueue_fetcher Tags

queue priority fetching priority-queue-based fetching fetching system fetch function

pqueue_fetcher Description

Implements a priority-queue-based fetching system pqueue_fetcher implements a priority-queue-based fetching system.The Fetcher class can be used by passing a fetch function which accepts locations of an arbitrary type.Locations can then be added via the fetcher's add method.It is meant to manage fetches by interrupting timed-out calls to fetch and by re-entering failed fetches back into the queue with an altered priority.Fetches which fail, either due to timeout or by virtue of being filtered out by a passed success function, are routed back into the source queue with their priority altered by another passed function.Succeeding fetches are passed on to the results queue.The fetch function may be any function which accepts a single parameter: thus the scope of tasks it may perform is relatively unrestricted.The Fetcher constructor also accepts a numeric threadcount argument, which will determine the number of concurrent fetch functions to run.Each fetch call will be made in a newly spawned thread. This is done as a relatively simple way to allow interruption of the call using the C PyThreadState_SetAsyncExc function.PortabilityBecause of its use of CPython's underlying C API, this module is not portable to other Python implementations.IssuesDue to deficiencies in Python's threading facilities which the author has been unable to work around, problems may result if the passed fetch blocks on I/O.An attempt has been made to work around these issues; see the reactor function for details.Despite this attempt, the code is still markedly sketchy. Consider yourself warned.Of related note is the method test.FetcherTester.test_incorrect_fission. This test highlights the fragility of the I/O block workaround. See the documentation for that method for more information. Requirements: · Python

pqueue_fetcher Related Software