Understanding SEO from Scratch (III): What Data is Needed to Assess the SEO Status of a Website?

After obtaining the valid page data, the next step is to gather the indexing data and calculate the indexing ratio.
The indexing ratio is primarily used to observe whether the website has crawling issues or content quality problems, serving as a basis for making further SEO optimization decisions.
How to obtain the indexing data of a website? There are two main methods:
For small websites with fewer pages: The URL data of pages indexed by Google can be obtained through Google Search Console (GSC), but this tool only provides up to 1,000 rows of data, making it suitable only for small websites.

For medium-sized websites with a large number of valid pages, this can be achieved through automated scripts that verify the indexing status of each valid page URL and obtain the indexing data.

In cases of a low indexing ratio for valid pages, further diagnosis is necessary to determine whether the issue is related to crawling or a problem with content quality. To determine if it is a crawling issue, data from the past three months of spider crawl activity needs to be obtained. So, how can we acquire the spider crawl data for the past three months?

There are two methods to obtain this data:
1) Through the GSC (Google Search Console) backend to get spider crawl data.
The advantage is that the statistical data is directly presented in the backend, such as the proportion of pages with different status codes and the overall crawling trend. However, it can only download up to 1,000 rows of data, which is suitable only for new or small websites.

2) Download server access logs for filtering and statistics.
You can log into the server, download the user access logs, and filter out the log data for Googlebot (the Google spider). Then, you can calculate the crawling data for all valid pages in the recent period (e.g., within 90 days).
Since this method involves some log cleaning techniques, it may not be suitable for beginners. There are some third-party tools available in the market to help with spider crawl data statistics, but according to Binomialtree SEO, these tools have significant limitations.
Here, Binomialtree SEO recommends using two Linux commands to filter and extract data from raw log files: awk and grep. These two commands can easily and quickly filter out any desired fields and information from the log files. Since these commands are executed in a Linux environment, Windows users can install the Cygwin64 tool to simulate a Linux environment and run these commands.
The use of these tools requires a detailed explanation, so I’ll keep it brief for now. For those who want to learn more, you can explore and learn about Cygwin64 and the awk/grep commands. Later, I will write a dedicated tutorial to show how to use Cygwin64 to filter the desired spider crawl data from complex log files.

After understanding the website’s indexing and crawling data to assess the indexing status and identify potential issues, the next step is to examine the coverage of target keywords on the website.
Target keyword coverage, literally, refers to how many of the target keywords you want to optimize are carried on specific pages of the website (this can be simply understood as the title containing the target keyword and the page having content relevant to that keyword). Traffic comes from keywords that rank, and for a page to rank in search results, it must have the target keyword. Therefore, checking keyword coverage is a crucial data point for fully understanding a website’s SEO status.

How to Check the Target Keyword Coverage of a Website?
To understand the target keyword coverage of a website, you need two sets of keyword data: the target keywords you want to optimize and the target keywords that the website has already covered.
1) Target Keywords to Optimize
You can expand a list of keywords based on competitors, industry trends, product needs, etc., by using third-party tools. Then, filter out the keywords you need according to your business goals. Some useful third-party tools for keyword research include Ahrefs, SEMrush, and KWFinder.
2) Keywords Already Covered by the Website
You can obtain this data by using the valid page URL data, which will give you information about the page titles and keywords. If feasible, technical staff can be asked to export this data based on the URLs, or you can manually crawl the front-end data for the URLs (a common method used by Binomialtree SEO). If a website does not directly provide keyword data, you may extract relevant keywords from the page titles. To simplify the process, you can assume that any page where the target keyword appears in the title is considered to have that keyword covered.

After obtaining the target keywords to optimize and the target keywords already covered by the website, you can clearly understand the website’s overall target keyword coverage by comparing these two sets of data. Once you fully understand the coverage of target keywords, it will be clear whether you need to add more pages or optimize existing ones to improve coverage, or if the coverage is already complete and no optimization is needed.

Organic traffic data is a direct reflection of SEO effectiveness. The most commonly used traffic analytics tool for foreign trade independent sites is Google Analytics (GA). It not only tracks user traffic and conversion data but also allows for custom filtering as needed, making it very convenient and easy to use, plus it’s free.
Similarly, for conversion data (whether it’s lead generation or sales), GA can be used to set up conversion goals and events to track conversion performance. By segmenting, comparing, and tracing the conversion data, you can more clearly analyze conversion issues and make targeted optimizations. Besides analyzing the issues, understanding the conversion data of SEO organic traffic also provides a clearer view of the value that the SEO channel brings to the company, serving as a data basis for future investments in SEO channels.