How to Compare Crawls in Screaming Frog
To use compare mode, simply switch the 'Mode' from Spider to List and upload either an XML sitemap or sitemap index. You can then filter by 'Response Code' and identify any redirects.
This can be a really useful way of checking for changes between crawls. The right-hand overview and issues tabs will populate to show current and previous data and identify changes ('added' and'removed'). This is great for identifying progress and tackling problems.
Page Speed
The Screaming Frog SEO spider works in a similar way to spiders used by search engines. It reads the HTML code of the starting page, identifies elements such as headings, images, and meta descriptions, and then follows internal links to identify other pages on the website and review them in the same way. This provides SEO specialists with a huge amount of data they can use to fix problems on websites and improve their rankings.
By default, Screaming Frog uses your computer’s RAM to store the information it’s exploring. This makes it incredibly fast, but can lead to problems when you’re crawling large sites. Thankfully, switching to database storage is simple and, as long as your machine has an SSD, can be much faster than RAM.
You can also integrate Screaming Frog with APIs like Google Analytics, Search Console, Moz, Ahrefs, and PageSpeed Insights. This enables you to add extra data to your audits, such as page visits or bounce rates.
SEO Company ScotlandAnother useful feature of Screaming Frog is its ability to identify URLs with duplicate content. This can be especially important for larger websites, where there may be multiple versions of a page. For example, some sites have a login and check out section that is secured with HTTPS, while the rest of the site is open over HTTP. Screaming Frog can scan for duplicates and report on them, helping you improve the user experience of your website.
Crawl Time
Crawling a website can tell you a lot about its structure and issues. It can also help you identify opportunities for improvement. But there are a few things you need to keep in mind when crawling a large site.
One of the most important factors is the amount of RAM your computer has. The more RAM you have, the more pages Screaming Frog can process during a scan. If you have limited RAM, it may be necessary to break up the site into multiple smaller crawls. Using an extension such as The Great Suspender can also help reduce RAM usage.
Another option is to use Screaming Frog’s built-in XML sitemap support. This feature allows you to compare the URLs in a sitemap with the URLs in Screaming Frog’s internal scan results. This can be useful for sites that are migrating to a new domain or for site audits.
In addition to XML sitemap support, Screaming Frog can also connect with other tools like PageSpeed Insights and Ahrefs to provide additional data in the results. This can be helpful in identifying issues such as slow loading pages and broken links. You can even connect your Google Analytics account to get a more complete picture of your site’s performance. You can find this information in the 'External' tab of Screaming Frog’s results.
Crawl Distance
In comparison mode, you can set a range of elements and metrics you wish to identify changes in. The more options you select, the longer it will take to complete a crawl. Once the compare analysis has been performed a new 'Change Detection' tab is displayed within the master view and Overview tab. The 'Compare' settings are also shown within the 'Config > Compare' menu.
This is a count of URLs that were found but weren't found in the previous crawl, indicating a change in page content. This is useful when analyzing site migrations or new website launches.
The 'Over 55 Characters' count shows the number of titles that exceed the 512px cutoff (a commonly used measure for title truncation). While it doesn't affect rankings this can impact click-through rates, so having an understanding of your title lengths is important.
Using database storage (as opposed to memory storage) allows you to recover crawls that have crashed during a scan and also access queued URLs. However, it's worth ensuring that the maximum allocation of RAM is less than your computer's total available RAM, as large site crawls can cause Screaming Frog to freeze once it reaches this limit. This will avoid any potential crashing issues during the scan. A large site can also take a long time to scan due to the high amount of data that must be processed by the spider.
Crawl Type
Depending on the size of your website and the amount of RAM your computer has, you may need to change your crawl type. By default, Screaming Frog uses your machine’s RAM to store data which makes it super fast but can be limiting when crawling large websites. To change this, go to Configuration > Storage Mode and select database storage (recommended for larger websites).
Another setting you can change is the user agent used by Screaming Frog. You can choose to use the Googlebot or a custom user agent to mimic how search engines see your website. This can be useful for comparing how your site performs to your competitors’.
You can also identify duplicate content on your website by filtering for Exact Duplicates or Near Duplicates in the Content tab. This can help you identify pages that are targeting the same keywords and confuse search engines, which could negatively impact your page rankings.
Lastly, Screaming Frog can help you verify that your Google Analytics code is being correctly implemented on your site. This is done by scanning a page to find all the URLs that have an Analytics tracking tag within it. You can then check the URLs in the Linked Pages tab to make sure they are being followed. This is an important step in ensuring that your analytics is accurate. You can also use this data to monitor how your site’s performance improves over time.
Crawl Size
The crawl size refers to how much RAM (Random Access Memory) the software uses during a scan. The higher this number, the more pages Screaming Frog can scan and analyse simultaneously.
Depending on the size of your website, this may be important to consider. A large website can take up a lot of RAM and slow down your computer when scanning. In this case, it might be better to switch to database storage mode instead, which saves your data to a folder on your hard drive rather than in Screaming Frog's memory.
You can also set the maximum size of your RAM buffer in Screaming Frog, which helps avoid your computer crashing when it starts running out of RAM during a scan. However, it's recommended that you only allocate the amount of RAM that your computer can actually hold, as allowing it to use too much may cause problems with performance.
Another feature of Screaming Frog is its ability to identify broken links, which can impact search engine rankings and user experience. For example, broken images or links to non-existent pages can be a huge red flag for SEOs. The tool can also detect internal page titles and meta descriptions, as well as check whether any URLs contain Google Analytics code. This information is particularly useful when conducting a technical audit or analysing competitor sites. This information is shown in the 'Internal' tab of the results.
Crawl Date
Depending on the amount of information you are collecting during a crawl, it can start to eat up your computer’s RAM. This can lead to a slow UI and even crashing the software. The best solution is to use the RAM storage mode and make sure your machine has plenty of RAM. Adding RAM is inexpensive and is easy to install.
You can also break up large directories into smaller crawlable chunks by using Xpath, CSS path or regex extractors. This allows you to collect a more detailed data set for specific areas of your site, e.g., pages including a particular keyword that need to be cleaned up or optimized.
Screaming Frog is a great tool for analysing internal and external links. It will allow you to identify broken links on your website and can provide useful insights into the overall structure of your site. You can check for broken links by running a crawl and filtering the results for 404 or 403 responses.
Screaming Frog’s ‘Compare’ mode is extremely helpful in identifying changes to SEO issues and opportunities between two different crawls of the same website. You can select a number of elements and metrics to compare, including the index status, the number of internal or external links, hreflang tags, pagination, PDF files and last modified date. You can then run a ‘Compare Analysis’, which will present you with a list of the differences found within those selected filters.
Comments
Post a Comment