diff options
| author | luccioman <luccioman@users.noreply.github.com> | 2017-08-16 09:30:33 +0200 |
|---|---|---|
| committer | luccioman <luccioman@users.noreply.github.com> | 2017-08-16 09:30:33 +0200 |
| commit | 3f0446f14b89e6a502363dc516f82f7dbab330df (patch) | |
| tree | 58b19834cc4cdcb4c9e8e357cd13a5cd3db6dd2c /htroot/api | |
| parent | 9da75ac76d7886a1a3a807746394e0b5aadb06ca (diff) | |
Ensure proper synchronous robots entry retrieval on first check.
Previously, when checking for the first time the robots.txt policy on a
unknown host (not cached in the robots table), result was always empty
in the /getpageinfo_p.xml api and in the /CrawlCheck_p.html page. Next
calls returned however the correct information.
Diffstat (limited to 'htroot/api')
| -rw-r--r-- | htroot/api/getpageinfo_p.java | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/htroot/api/getpageinfo_p.java b/htroot/api/getpageinfo_p.java index 309421a63..8da9cef8a 100644 --- a/htroot/api/getpageinfo_p.java +++ b/htroot/api/getpageinfo_p.java @@ -221,7 +221,6 @@ public class getpageinfo_p { // determine if crawling of the current URL is allowed ClientIdentification.Agent agent = ClientIdentification.getAgent(post.get("agentName", ClientIdentification.yacyInternetCrawlerAgentName)); - sb.robots.ensureExist(theURL, agent, true); RobotsTxtEntry robotsEntry = sb.robots.getEntry(theURL, agent); prop.put("robots-allowed", robotsEntry == null ? 1 : robotsEntry.isDisallowed(theURL) ? 0 : 1); prop.putHTML("robotsInfo", robotsEntry == null ? "" : robotsEntry.getInfo()); |