.Google's Gary Illyes shows Googlebot collects links as opposed to "adhering to" them, negating formal documents.Googlebot collects web links rather than following all of them in real-time.Gary Illyes discussed this info in Google's 'Explore Off The Document' podcast.Google.com's records negates the actual spider behavior.