I recommend looking for those with 3xx, 4xx, and 5xx status codes because you want to see redirected or error pages that you serve to crawlers. Error and redirected pages for Googlebot [Click to enlarge] From there, you can identify the top pages generating the most redirects or errors. You can export the data and prioritize these pages to include in your SEO Shadow Making recommendations. 4. What are the main pages crawled by each of the search robots? Check if they coincide with the most important ones on your site. When searching for the search bot you want, you can directly select the “requestURI” filter to get a list of the main web documents, be it resources Shadow Making or pages, that the bot requests. You can consult them directly in the interface (to verify that they have an HTTP 200 status, for example) or export them to an Excel document, where you can determine if they coincide with your priority pages. Main crawled pages [Click to enlarge] If your most important.
Check that they are correctly accessing the Shadow Making relevant pages and resources in each case. I included this one specifically for websites that serve different content to users in different locations. In some cases, these websites unknowingly provide a Shadow Making poor experience for crawlers with IP addresses from other countries - blocking them outright or allowing them to access only one version of the content (preventing them from explore other versions). Google now supports locale-aware crawling to discover content specifically intended to target other Shadow Making countries, but it's still a good idea to make sure all your content is being crawled. Otherwise, it may indicate that your website is not set up correctly. After segmenting by user agent, you can then filter by IP to verify that the site is serving the correct version of each page to crawlers from the relevant countries. Googlebot's IP [Click to enlarge] For example, look what happens when I try to access the NBA site with a Spanish IP address.
If you don't already have one). 'experience): Excel If you're familiar with Excel - if creating PivotTables and using VLOOKUP is second nature to you - you might want to give Excel a try by following the steps given in this guide by BuiltVisible. It's also important to note that even if you use any of the other options in the tool, at some point you'll need to export the data you've collected into Excel. This will produce the data in a format that is easy to integrate or compare with other data Shadow Making sources such as Google Analytics or Google Search Console. Whether to use Excel throughout the analysis or only at the end will depend on how much time you want to Shadow Making spend using it to filter, segment, and organize the data. Open source tools It's your choice if you don't have a budget to spend on tools but have the technical resources to set them up. The most popular open source option is Elastic's ELK stack, which includes Kibana, Elasticsearch, and Logstash. Paid tools It's really the best option.