Skip to content

Commit 0010c13

Browse files
sjdirectsjdirect
committed
added new abot config value for x-robot-tag http header [skip ci]
1 parent 1b5feb3 commit 0010c13

File tree

1 file changed

+1
-0
lines changed

1 file changed

+1
-0
lines changed

‎README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ using Abot.Poco;
9797
<politeness
9898
isRespectRobotsDotTextEnabled="false"
9999
isRespectMetaRobotsNoFollowEnabled="false"
100+
isRespectHttpXRobotsTagHeaderNoFollowEnabled="false"
100101
isRespectAnchorRelNoFollowEnabled="false"
101102
isIgnoreRobotsDotTextIfRootDisallowedEnabled="false"
102103
robotsDotTextUserAgentString="abot"

0 commit comments

Comments
 (0)