public class RobotsManager extends BaseTable
Field | Type | Description |
---|---|---|
hostname | VARCHAR(255) | Primary Key |
robotsdata | BIGINT | |
expirationtime | BLOB |
Modifier and Type | Class and Description |
---|---|
protected static class |
RobotsManager.HostDescription
This is the object description for a robots host object.
|
protected static class |
RobotsManager.HostExecutor
This is the executor object for locating robots host objects.
|
protected static class |
RobotsManager.Record
This class represents a record in a robots.txt file.
|
protected static class |
RobotsManager.RobotsCacheClass
Cache class for robots.
|
protected static class |
RobotsManager.RobotsData
This is a cached data item.
|
Modifier and Type | Field and Description |
---|---|
static String |
_rcsid |
protected static String |
expirationField |
protected static String |
hostField |
protected static RobotsManager.RobotsCacheClass |
robotsCacheClass |
protected static String |
robotsField |
dbInterface, tableName
Constructor and Description |
---|
RobotsManager(IThreadContext tc,
IDBInterface database)
Constructor.
|
Modifier and Type | Method and Description |
---|---|
Boolean |
checkFetchAllowed(String userAgent,
String hostName,
long currentTime,
String pathString,
IVersionActivity activities)
Read robots.txt data from the cache or from the database.
|
void |
deinstall()
Uninstall the manager.
|
protected static boolean |
doesPathMatch(String path,
int pathIndex,
String spec,
int specIndex)
Recursive method for matching specification to path.
|
protected static boolean |
doesPathMatch(String path,
String spec)
Check if path matches specification
|
protected static String |
getRobotsKey(String hostName)
Construct a key which represents an individual host name.
|
void |
install()
Install the manager.
|
protected static String |
makeReadable(String inputString)
Convert a string from the robots file into a readable form that does NOT contain NUL characters (since postgresql does not accept those).
|
protected RobotsManager.RobotsData |
readRobotsData(String hostName,
IVersionActivity activities)
Read robots data, if it exists.
|
void |
writeRobotsData(String hostName,
long expirationTime,
InputStream data)
Write robots.txt, replacing any existing row.
|
addTableIndex, analyzeTable, beginTransaction, buildConjunctionClause, constructCountClause, constructDistinctOnClause, constructDoubleCastClause, constructOffsetLimitClause, constructRegexpClause, constructSubstringClause, endTransaction, findConjunctionClauseMax, getDatabaseCacheKey, getDBInterface, getMaxInClause, getMaxOrClause, getSleepAmt, getTableIndexes, getTableName, getTableSchema, getTransactionID, getWindowedReportMaxRows, makeTableKey, noteModifications, performAddIndex, performAlter, performCommit, performCreate, performDelete, performDrop, performInsert, performModification, performQuery, performQuery, performRemoveIndex, performUpdate, prepareRowForSave, readRow, reindexTable, signalRollback, sleepFor
public static final String _rcsid
protected static RobotsManager.RobotsCacheClass robotsCacheClass
protected static final String hostField
protected static final String robotsField
protected static final String expirationField
public RobotsManager(IThreadContext tc, IDBInterface database) throws ManifoldCFException
tc
- is the thread context.database
- is the database handle.ManifoldCFException
public void install() throws ManifoldCFException
ManifoldCFException
public void deinstall() throws ManifoldCFException
ManifoldCFException
public Boolean checkFetchAllowed(String userAgent, String hostName, long currentTime, String pathString, IVersionActivity activities) throws ManifoldCFException
hostName
- is the host for which the data is desired.currentTime
- is the time of the check.ManifoldCFException
public void writeRobotsData(String hostName, long expirationTime, InputStream data) throws ManifoldCFException, IOException
hostName
- is the host.expirationTime
- is the time this data should expire.data
- is the robots data stream. May be null.ManifoldCFException
IOException
protected static String getRobotsKey(String hostName)
hostName
- is the name of the connector.protected RobotsManager.RobotsData readRobotsData(String hostName, IVersionActivity activities) throws ManifoldCFException
ManifoldCFException
protected static String makeReadable(String inputString)
protected static boolean doesPathMatch(String path, String spec)