Differences
This shows you the differences between two versions of the page.
| Both sides previous revision Previous revision Next revision | Previous revision | ||
| proxy_scraper:url_metrics [2018-06-17 17:36] – [URL Metrics Scanner] sven | proxy_scraper:url_metrics [2020-05-12 14:52] (current) – [SEORank (needs API-Key)] sven | ||
|---|---|---|---|
| Line 13: | Line 13: | ||
| After resolving the data, you can export or filter them. | After resolving the data, you can export or filter them. | ||
| - | Please note that unlike other tools, **this module does not require any accounts** to get the data. All you need are proxies which the program | + | Here is a list of all possible metrics |
| - | At startup it might be a bit slow as the software has to find proxies that would work. But once found they get cached and processing should be faster. Even after a program restart it should be faster. | + | ===== Metrics accessible without any API-Key ===== |
| - | Here is a list of all possible metrics you can extract: | ||
| ^Column^Description^ | ^Column^Description^ | ||
| - | |Title|The title of the page, if available| | ||
| |Yandex TIC URL|Yandex TIC for URL| | |Yandex TIC URL|Yandex TIC for URL| | ||
| |Yandex TIC SubDomain|Yandex TIC for SubDomain| | |Yandex TIC SubDomain|Yandex TIC for SubDomain| | ||
| |Yandex TIC RootDomain|Yandex TIC for RootDomain| | |Yandex TIC RootDomain|Yandex TIC for RootDomain| | ||
| - | |SEMRush Rank-SubDomain|SEMRush Rank for SubDomain| | + | |Yandex AGS SubDomain|Yandex TIC for SubDomain| |
| - | |SEMRush Rank-RootDomain|SEMRush Rank for RootDomain| | + | |Yandex AGS RootDomain|Yandex TIC for RootDomain| |
| - | |SEMRush Cost-SubDomain|how much need to spend for get same number of visitors from PPC for SubDomain| | + | |
| - | |SEMRush Cost-RootDomain|how much need to spend for get same number of visitors from PPC for RootDomain| | + | |
| - | |SEMRush Traffic-SubDomain|estimated number of visitors coming from search engines for SubDomain| | + | |
| - | |SEMRush Traffic-RootDomain|estimated number of visitors coming from search engines for RootDomain| | + | |
| - | |SEMRush Keywords-SubDomain|number of ranked keywords in google ranking for SubDomain| | + | |
| - | |SEMRush Keywords-RootDomain|number of ranked keywords in google ranking | + | |
| |Alexa Traffic Rank|Alexa Traffic Rank| | |Alexa Traffic Rank|Alexa Traffic Rank| | ||
| |WebArchive Date|Oldest Webarchive Date| | |WebArchive Date|Oldest Webarchive Date| | ||
| - | |Facebook Shares|Number of Facebook Shares| | ||
| - | |Facebook Comments|Number of Facebook Comments| | ||
| - | |Google+ Shares|Number of Google+ Shares| | ||
| |URL Status|Informations about the URL| | |URL Status|Informations about the URL| | ||
| |Google Indexed Pages|How many URLs are indexed for this SubDomain| | |Google Indexed Pages|How many URLs are indexed for this SubDomain| | ||
| + | |Google Indexed URL|Is this URLs indexed in Google?| | ||
| + | |Country|Country that this website is for| | ||
| + | |Language|Language that this website is in| | ||
| - | There are many other Metric Items you can choose | + | ===== Metrics accessible only via PR Emulation Software ===== |
| + | |||
| + | ^Column^Description^ | ||
| + | |PR URL|PageRank by Google for URL| | ||
| + | |PR SubDomain|PageRank by Google for SubDomain| | ||
| + | |PR RootDomain|PageRank by Google for RootDomain| | ||
| + | |||
| + | ===== Majestic (requires API-key) ===== | ||
| + | You can get an API-Key on the [[https:// | ||
| + | ^Column^Description^ | ||
| + | |TrustFlow URL|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
| + | |TrustFlow SubDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
| + | |TrustFlow RootDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
| + | |CitationFlow URL|Number of predicting how influential a URL might be based on how many sites link to it| | ||
| + | |CitationFlow SubDomain|Number of predicting how influential a URL might be based on how many sites link to it| | ||
| + | |CitationFlow RootDomain|Number of predicting how influential a URL might be based on how many sites link to it| | ||
| + | |ExtBackLinks URL|Number of backlinks to this URL| | ||
| + | |ExtBackLinks SubDomain|Number of backlinks to this SubDomain| | ||
| + | |ExtBackLinks RootDomain|Number of backlinks to this RootDomain| | ||
| + | |RefDomains URL|Number of unique domains linking to this URL| | ||
| + | |RefDomains SubDomain|Number of unique domains linking to this SubDomain| | ||
| + | |RefDomains RootDomain|Number of unique domains linking to this RootDomain| | ||
| + | |ACRank RootDomain|CitationFlow like Rank (but less relevant)| | ||
| + | |ACRank SubDomain|CitationFlow like Rank (but less relevant)| | ||
| + | |ACRank URL|CitationFlow like Rank (but less relevant)| | ||
| + | |CrawledFlag RootDomain|Was it actually crawled| | ||
| + | |CrawledFlag SubDomain|Was it actually crawled| | ||
| + | |CrawledFlag URL|Was it actually crawled| | ||
| + | |CrawledURLs RootDomain|| | ||
| + | |CrawledURLs SubDomain|| | ||
| + | |CrawledURLs URL|| | ||
| + | |ExtBackLinksEDU RootDomain|Number of external .EDU/ | ||
| + | |ExtBackLinksEDU SubDomain|Number of external .EDU/ | ||
| + | |ExtBackLinksEDU URL|Number of external .EDU/ | ||
| + | |ExtBackLinksGOV RootDomain|Number of external .GOV/ | ||
| + | |ExtBackLinksGOV SubDomain|Number of external .GOV/ | ||
| + | |ExtBackLinksGOV URL|Number of external .GOV/ | ||
| + | |FinalRedirectResult RootDomain|Describes the response | ||
| + | |FinalRedirectResult SubDomain|Describes the response from attempting to crawl the page after all redirects are followed| | ||
| + | |FinalRedirectResult URL|Describes the response from attempting to crawl the page after all redirects are followed| | ||
| + | |IndexedURLs RootDomain|Amount of indexted URLs| | ||
| + | |IndexedURLs SubDomain|Amount of indexted URLs| | ||
| + | |IndexedURLs URL|Amount of indexted URLs| | ||
| + | |Language RootDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
| + | |Language SubDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
| + | |Language URL|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
| + | |LanguageConfidence RootDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
| + | |LanguageConfidence SubDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
| + | |LanguageConfidence URL|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
| + | |LanguageDesc RootDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
| + | |LanguageDesc SubDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
| + | |LanguageDesc URL|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
| + | |LanguagePageRatios RootDomain|| | ||
| + | |LanguagePageRatios SubDomain|| | ||
| + | |LanguagePageRatios URL|| | ||
| + | |LanguageTotalPages RootDomain|| | ||
| + | |LanguageTotalPages SubDomain|| | ||
| + | |LanguageTotalPages URL|| | ||
| + | |LastCrawlDate RootDomain|Date when it was crawled| | ||
| + | |LastCrawlDate SubDomain|Date when it was crawled| | ||
| + | |LastCrawlDate URL|Date when it was crawled| | ||
| + | |LastCrawlResult RootDomain|Status code when being crawled| | ||
| + | |LastCrawlResult SubDomain|Status code when being crawled| | ||
| + | |LastCrawlResult URL|Status code when being crawled| | ||
| + | |LastSeen RootDomain|| | ||
| + | |LastSeen SubDomain|| | ||
| + | |LastSeen URL|| | ||
| + | |NonUniqueLinkTypeDeleted RootDomain|| | ||
| + | |NonUniqueLinkTypeDeleted SubDomain|| | ||
| + | |NonUniqueLinkTypeDeleted URL|| | ||
| + | |NonUniqueLinkTypeFrame RootDomain|| | ||
| + | |NonUniqueLinkTypeFrame SubDomain|| | ||
| + | |NonUniqueLinkTypeFrame URL|| | ||
| + | |NonUniqueLinkTypeHomepages RootDomain|| | ||
| + | |NonUniqueLinkTypeHomepages SubDomain|| | ||
| + | |NonUniqueLinkTypeHomepages URL|| | ||
| + | |NonUniqueLinkTypeImageLink RootDomain|| | ||
| + | |NonUniqueLinkTypeImageLink SubDomain|| | ||
| + | |NonUniqueLinkTypeImageLink URL|| | ||
| + | |NonUniqueLinkTypeIndirect RootDomain|| | ||
| + | |NonUniqueLinkTypeIndirect SubDomain|| | ||
| + | |NonUniqueLinkTypeIndirect URL|| | ||
| + | |NonUniqueLinkTypeNoFollow RootDomain|| | ||
| + | |NonUniqueLinkTypeNoFollow SubDomain|| | ||
| + | |NonUniqueLinkTypeNoFollow URL|| | ||
| + | |NonUniqueLinkTypeProtocolHTTPS RootDomain|| | ||
| + | |NonUniqueLinkTypeProtocolHTTPS SubDomain|| | ||
| + | |NonUniqueLinkTypeProtocolHTTPS URL|| | ||
| + | |NonUniqueLinkTypeRedirect RootDomain|| | ||
| + | |NonUniqueLinkTypeRedirect SubDomain|| | ||
| + | |NonUniqueLinkTypeRedirect URL|| | ||
| + | |NonUniqueLinkTypeTextLink RootDomain|| | ||
| + | |NonUniqueLinkTypeTextLink SubDomain|| | ||
| + | |NonUniqueLinkTypeTextLink URL|| | ||
| + | |OutDomainsExternal RootDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutDomainsExternal SubDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutDomainsExternal URL|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksExternal RootDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksExternal SubDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksExternal URL|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksInternal RootDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksInternal SubDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksInternal URL|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
| + | |OutLinksPages RootDomain|| | ||
| + | |OutLinksPages SubDomain|| | ||
| + | |OutLinksPages URL|| | ||
| + | |RedirectFlag RootDomain|was it redirecting| | ||
| + | |RedirectFlag SubDomain|was it redirecting| | ||
| + | |RedirectFlag URL|was it redirecting| | ||
| + | |RedirectTo RootDomain|Final redirecting URL| | ||
| + | |RedirectTo SubDomain|Final redirecting URL| | ||
| + | |RedirectTo URL|Final redirecting URL| | ||
| + | |RefDomainsEDU RootDomain|Number of referring .EDU/ | ||
| + | |RefDomainsEDU SubDomain|Number of referring .EDU/ | ||
| + | |RefDomainsEDU URL|Number of referring .EDU/ | ||
| + | |RefDomainsGOV RootDomain|Number of referring .GOV/ | ||
| + | |RefDomainsGOV SubDomain|Number of referring .GOV/ | ||
| + | |RefDomainsGOV URL|Number of referring .GOV/ | ||
| + | |RefDomainTypeDirect RootDomain|| | ||
| + | |RefDomainTypeDirect SubDomain|| | ||
| + | |RefDomainTypeDirect URL|| | ||
| + | |RefDomainTypeFollow RootDomain|| | ||
| + | |RefDomainTypeFollow SubDomain|| | ||
| + | |RefDomainTypeFollow URL|| | ||
| + | |RefDomainTypeHomepageLink RootDomain|| | ||
| + | |RefDomainTypeHomepageLink SubDomain|| | ||
| + | |RefDomainTypeHomepageLink URL|| | ||
| + | |RefDomainTypeLive RootDomain|| | ||
| + | |RefDomainTypeLive SubDomain|| | ||
| + | |RefDomainTypeLive URL|| | ||
| + | |RefDomainTypeProtocolHTTPS RootDomain|| | ||
| + | |RefDomainTypeProtocolHTTPS SubDomain|| | ||
| + | |RefDomainTypeProtocolHTTPS URL|| | ||
| + | |RefIPs RootDomain|Number of referring IP addresses pointing to this URL| | ||
| + | |RefIPs SubDomain|Number of referring IP addresses pointing to this URL| | ||
| + | |RefIPs URL|Number of referring IP addresses pointing to this URL| | ||
| + | |RefLanguage RootDomain|| | ||
| + | |RefLanguage SubDomain|| | ||
| + | |RefLanguage URL|| | ||
| + | |RefLanguageConfidence RootDomain|| | ||
| + | |RefLanguageConfidence SubDomain|| | ||
| + | |RefLanguageConfidence URL|| | ||
| + | |RefLanguageDesc RootDomain|| | ||
| + | |RefLanguageDesc SubDomain|| | ||
| + | |RefLanguageDesc URL|| | ||
| + | |RefLanguagePageRatios RootDomain|| | ||
| + | |RefLanguagePageRatios SubDomain|| | ||
| + | |RefLanguagePageRatios URL|| | ||
| + | |RefLanguageTotalPages RootDomain|| | ||
| + | |RefLanguageTotalPages SubDomain|| | ||
| + | |RefLanguageTotalPages URL|| | ||
| + | |RefSubNets RootDomain|Number of referring IP Class C subnets pointing to this URL| | ||
| + | |RefSubNets SubDomain|Number of referring IP Class C subnets pointing to this URL| | ||
| + | |RefSubNets URL|Number of referring IP Class C subnets pointing to this URL| | ||
| + | |RootDomainIPAddress RootDomain|| | ||
| + | |RootDomainIPAddress SubDomain|| | ||
| + | |RootDomainIPAddress URL|| | ||
| + | |TotalNonUniqueLinks RootDomain|| | ||
| + | |TotalNonUniqueLinks SubDomain|| | ||
| + | |TotalNonUniqueLinks URL|| | ||
| + | |TrustMetric RootDomain|| | ||
| + | |TrustMetric SubDomain|| | ||
| + | |TrustMetric URL|| | ||
| + | |||
| + | ===== DomDetailer (requires API-Key) ===== | ||
| + | You can get an API-Key on the [[https:// | ||
| + | |||
| + | ^Column^Description^ | ||
| + | |DD MozLinks|The number of links (equity or nonequity or not, internal or external) to the URL| | ||
| + | |DD MozPage Authority|A normalized 100-point score representing the likelihood of a page to rank well in search engine results| | ||
| + | |DD MozDomain Authority|A normalized 100-point score representing the likelihood of a domain to rank well in search engine results| | ||
| + | |DD MozRank|The MozRank of the URL, in the normalized 10-point score| | ||
| + | |DD MozTrust|The MozTrust of the URL, in the normalized 10-point| | ||
| + | |DD MozSpam|Bit field of triggered spam flags| | ||
| + | |DD Facebook Comments|Facebook Comments| | ||
| + | |DD Facebook Shares|Facebook Shares| | ||
| + | |DD Google+ Shares|Google+ shares| | ||
| + | |DD Pinterest Pins|pinterest shares| | ||
| + | |DD LinkedIn Shares|LinkedIn shares| | ||
| + | |DD majesticLinks|Number of backlinks to this URL| | ||
| + | |DD majesticRefDomains|Number of unique domains linking to this URL| | ||
| + | |DD majesticRefDomainsEDU|Number of unique .edu domains linking to this URL| | ||
| + | |DD majesticRefDomainsGOV|Number of unique .gov domains linking to this URL| | ||
| + | |DD majesticRefSubnets|Number of unique subnets linking to this URL| | ||
| + | |DD majesticTrustFlow|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
| + | |DD majesticCitationFlow|Number of predicting how influential a URL might be based on how many sites link to it| | ||
| + | |DD majesticCat|Suggested Category of this site.| | ||
| + | |||
| + | ===== SEORank (needs API-Key) ===== | ||
| + | You can get an API-Key on the [[https:// | ||
| + | |||
| + | ^Column^Description^ | ||
| + | |SEORank Domain Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how DOMAIN will be ranked on search engines, DA linked with link counts, MozRank, MozTrust scores and other data.| | ||
| + | |SEORank Page Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how URL will be ranked on search engines, PA linked with link counts, MozRank, MozTrust and other data.| | ||
| + | |SEORank Moz Rank|represents a link popularity score that showing the importance of given web page on the Internet.| | ||
| + | |SEORank Links In|number of links (equity or nonequity or not, internal or external) to the URL (data provided by Moz).| | ||
| + | |SEORank External Equity Links|number of external equity links to the URL (data provided by Moz).| | ||
| + | |SEORank Alexa Rank|global Alexa rank of domain.| | ||
| + | |SEORank ALexa Links number|number of links to domain.| | ||
| + | |SEORank Alexa Country|ISO2 code of country where domain is the most popular.| | ||
| + | |SEORank Alexa Country Rank|Alexa Rank of domain in country where it is popular.| | ||
| + | |SEORank SemRush Domain|domain that was taken by SemRush from url for analysis.| | ||
| + | |SEORank SemRush Rank|rank of domain according to SemRush.| | ||
| + | |SEORank SemRush Keywords number|number of keywords where site in Google| | ||
| + | |SEORank SemRush Traffic|number of users expected to visit the website during the following month.| | ||
| + | |SEORank SemRush Costs|estimated price of organic keywords in Google AdWords.| | ||
| + | |SEORank SemRush URL Links number|number of links to URL according to SemRush.| | ||
| + | |SEORank SemRush HOSTNAME Links number|number of links to HOSTNAME.| | ||
| + | |SEORank SemRush DOMAIN Links number|number of links to SemRush Domain .| | ||
| + | |SEORank Spam Score|number from 0 to 17, highest number means highest percent of sites that contains link to url but penalized or banned by Google.| | ||
| + | |SEORank Citation Flow|number from 0 to 100 (higher is better), it displaying how influential a URL might be based on how many sites link to it.| | ||
| + | |SEORank Trust Flow|number from 0 to 100 (higher is better), it displaying how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors.| | ||
| + | |SEORank ExtBackLinks|number of external backlinks to current URL (data provided by Majestic).| | ||
| + | |SEORank Refered Domains|number of domains with pages that refered to url| | ||
| ====== Google-PR Emulation ====== | ====== Google-PR Emulation ====== | ||