.Google.com's John Mueller answered a question about why Google marks web pages that are actually disallowed coming from crawling through robots.txt and why the it is actually secure to ignore the similar Browse Console reports regarding those crawls.Crawler Website Traffic To Query Guideline URLs.The individual asking the inquiry recorded that crawlers were actually generating links to non-existent query guideline Links (? q= xyz) to webpages along with noindex meta tags that are likewise obstructed in robots.txt. What motivated the concern is actually that Google.com is creeping the links to those webpages, obtaining obstructed through robots.txt (without envisioning a noindex robotics meta tag) then acquiring turned up in Google.com Search Console as "Indexed, though obstructed by robots.txt.".The individual asked the following concern:." But right here is actually the significant inquiry: why would Google.com mark web pages when they can not also find the web content? What's the benefit during that?".Google.com's John Mueller verified that if they can not creep the webpage they can not find the noindex meta tag. He additionally helps make an interesting acknowledgment of the site: hunt driver, encouraging to ignore the results because the "ordinary" customers won't see those end results.He wrote:." Yes, you're right: if our company can not crawl the web page, our team can't observe the noindex. That said, if our team can not crawl the webpages, then there is actually not a whole lot for our team to index. So while you may find a few of those webpages with a targeted website:- inquiry, the typical individual won't view them, so I would not fuss over it. Noindex is actually additionally alright (without robots.txt disallow), it simply means the Links will end up being crawled (and find yourself in the Explore Console record for crawled/not catalogued-- neither of these statuses trigger issues to the rest of the web site). The vital part is that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's solution validates the constraints in using the Internet site: hunt progressed search operator for diagnostic main reasons. One of those reasons is due to the fact that it is actually not linked to the regular search index, it's a distinct point entirely.Google.com's John Mueller talked about the web site search driver in 2021:." The short answer is actually that an internet site: query is actually not suggested to be complete, neither used for diagnostics functions.A web site question is actually a details type of search that confines the end results to a particular internet site. It's primarily just words web site, a colon, and afterwards the web site's domain.This question limits the end results to a specific website. It's certainly not indicated to be a comprehensive assortment of all the web pages coming from that internet site.".2. Noindex tag without making use of a robots.txt is actually great for these type of circumstances where a bot is linking to non-existent web pages that are getting found by Googlebot.3. Links along with the noindex tag are going to produce a "crawled/not recorded" entry in Search Console and that those will not have an unfavorable impact on the rest of the site.Check out the concern as well as address on LinkedIn:.Why will Google mark pages when they can not even view the content?Featured Picture by Shutterstock/Krakenimages. com.