Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
proxy_scraper:url_metrics [2017-02-13 22:37] – svenproxy_scraper:url_metrics [2020-05-12 14:52] (current) – [SEORank (needs API-Key)] sven
Line 13: Line 13:
 After resolving the data, you can export or filter them. After resolving the data, you can export or filter them.
  
-Please note that unlike other tools, **this module does not require any accounts** to get the data. All you need are proxies which the program is of course delivering to you.+Here is a list of all possible metrics you can extract:
  
-At startup it might be a bit slow as the software has to find proxies that would work. But once found they get cached and processing should be faster. Even after a program restart it should be faster.+===== Metrics accessible without any API-Key =====
  
-Here is a list of all possible metrics you can extract: 
 ^Column^Description^ ^Column^Description^
-|Title|The title of the page, if available| 
 |Yandex TIC URL|Yandex TIC for URL| |Yandex TIC URL|Yandex TIC for URL|
 |Yandex TIC SubDomain|Yandex TIC for SubDomain| |Yandex TIC SubDomain|Yandex TIC for SubDomain|
 |Yandex TIC RootDomain|Yandex TIC for RootDomain| |Yandex TIC RootDomain|Yandex TIC for RootDomain|
-|SEMRush Rank-SubDomain|SEMRush Rank for SubDomain| +|Yandex AGS SubDomain|Yandex TIC for SubDomain| 
-|SEMRush Rank-RootDomain|SEMRush Rank for RootDomain| +|Yandex AGS RootDomain|Yandex TIC for RootDomain|
-|SEMRush Cost-SubDomain|how much need to spend for get same number of visitors from PPC for SubDomain| +
-|SEMRush Cost-RootDomain|how much need to spend for get same number of visitors from PPC for RootDomain| +
-|SEMRush Traffic-SubDomain|estimated number of visitors coming from search engines for SubDomain| +
-|SEMRush Traffic-RootDomain|estimated number of visitors coming from search engines for RootDomain| +
-|SEMRush Keywords-SubDomain|number of ranked keywords in google ranking for SubDomain| +
-|SEMRush Keywords-RootDomain|number of ranked keywords in google ranking for RootDomain|+
 |Alexa Traffic Rank|Alexa Traffic Rank| |Alexa Traffic Rank|Alexa Traffic Rank|
 |WebArchive Date|Oldest Webarchive Date| |WebArchive Date|Oldest Webarchive Date|
-|Facebook Shares|Number of Facebook Shares| 
-|Facebook Comments|Number of Facebook Comments| 
-|Google+ Shares|Number of Google+ Shares| 
 |URL Status|Informations about the URL| |URL Status|Informations about the URL|
 |Google Indexed Pages|How many URLs are indexed for this SubDomain| |Google Indexed Pages|How many URLs are indexed for this SubDomain|
 +|Google Indexed URL|Is this URLs indexed in Google?|
 +|Country|Country that this website is for|
 +|Language|Language that this website is in|
  
 +===== Metrics accessible only via PR Emulation Software =====
  
 +^Column^Description^
 +|PR URL|PageRank by Google for URL|
 +|PR SubDomain|PageRank by Google for SubDomain|
 +|PR RootDomain|PageRank by Google for RootDomain|
 +
 +===== Majestic (requires API-key) =====
 +You can get an API-Key on the [[https://majestic.com/|Majestic Homepage]].
 +^Column^Description^
 +|TrustFlow URL|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors|
 +|TrustFlow SubDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors|
 +|TrustFlow RootDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors|
 +|CitationFlow URL|Number of predicting how influential a URL might be based on how many sites link to it|
 +|CitationFlow SubDomain|Number of predicting how influential a URL might be based on how many sites link to it|
 +|CitationFlow RootDomain|Number of predicting how influential a URL might be based on how many sites link to it|
 +|ExtBackLinks URL|Number of backlinks to this URL|
 +|ExtBackLinks SubDomain|Number of backlinks to this SubDomain|
 +|ExtBackLinks RootDomain|Number of backlinks to this RootDomain|
 +|RefDomains URL|Number of unique domains linking to this URL|
 +|RefDomains SubDomain|Number of unique domains linking to this SubDomain|
 +|RefDomains RootDomain|Number of unique domains linking to this RootDomain|
 +|ACRank RootDomain|CitationFlow like Rank (but less relevant)|
 +|ACRank SubDomain|CitationFlow like Rank (but less relevant)|
 +|ACRank URL|CitationFlow like Rank (but less relevant)|
 +|CrawledFlag RootDomain|Was it actually crawled|
 +|CrawledFlag SubDomain|Was it actually crawled|
 +|CrawledFlag URL|Was it actually crawled|
 +|CrawledURLs RootDomain||
 +|CrawledURLs SubDomain||
 +|CrawledURLs URL||
 +|ExtBackLinksEDU RootDomain|Number of external .EDU/.AC/.EDU.xx domains|
 +|ExtBackLinksEDU SubDomain|Number of external .EDU/.AC/.EDU.xx domains|
 +|ExtBackLinksEDU URL|Number of external .EDU/.AC/.EDU.xx domains|
 +|ExtBackLinksGOV RootDomain|Number of external .GOV/.MIL/.GOV.xx domains.|
 +|ExtBackLinksGOV SubDomain|Number of external .GOV/.MIL/.GOV.xx domains.|
 +|ExtBackLinksGOV URL|Number of external .GOV/.MIL/.GOV.xx domains.|
 +|FinalRedirectResult RootDomain|Describes the response from attempting to crawl the page after all redirects are followed|
 +|FinalRedirectResult SubDomain|Describes the response from attempting to crawl the page after all redirects are followed|
 +|FinalRedirectResult URL|Describes the response from attempting to crawl the page after all redirects are followed|
 +|IndexedURLs RootDomain|Amount of indexted URLs|
 +|IndexedURLs SubDomain|Amount of indexted URLs|
 +|IndexedURLs URL|Amount of indexted URLs|
 +|Language RootDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).|
 +|Language SubDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).|
 +|Language URL|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).|
 +|LanguageConfidence RootDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).|
 +|LanguageConfidence SubDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).|
 +|LanguageConfidence URL|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).|
 +|LanguageDesc RootDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English,German).|
 +|LanguageDesc SubDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English,German).|
 +|LanguageDesc URL|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English,German).|
 +|LanguagePageRatios RootDomain||
 +|LanguagePageRatios SubDomain||
 +|LanguagePageRatios URL||
 +|LanguageTotalPages RootDomain||
 +|LanguageTotalPages SubDomain||
 +|LanguageTotalPages URL||
 +|LastCrawlDate RootDomain|Date when it was crawled|
 +|LastCrawlDate SubDomain|Date when it was crawled|
 +|LastCrawlDate URL|Date when it was crawled|
 +|LastCrawlResult RootDomain|Status code when being crawled|
 +|LastCrawlResult SubDomain|Status code when being crawled|
 +|LastCrawlResult URL|Status code when being crawled|
 +|LastSeen RootDomain||
 +|LastSeen SubDomain||
 +|LastSeen URL||
 +|NonUniqueLinkTypeDeleted RootDomain||
 +|NonUniqueLinkTypeDeleted SubDomain||
 +|NonUniqueLinkTypeDeleted URL||
 +|NonUniqueLinkTypeFrame RootDomain||
 +|NonUniqueLinkTypeFrame SubDomain||
 +|NonUniqueLinkTypeFrame URL||
 +|NonUniqueLinkTypeHomepages RootDomain||
 +|NonUniqueLinkTypeHomepages SubDomain||
 +|NonUniqueLinkTypeHomepages URL||
 +|NonUniqueLinkTypeImageLink RootDomain||
 +|NonUniqueLinkTypeImageLink SubDomain||
 +|NonUniqueLinkTypeImageLink URL||
 +|NonUniqueLinkTypeIndirect RootDomain||
 +|NonUniqueLinkTypeIndirect SubDomain||
 +|NonUniqueLinkTypeIndirect URL||
 +|NonUniqueLinkTypeNoFollow RootDomain||
 +|NonUniqueLinkTypeNoFollow SubDomain||
 +|NonUniqueLinkTypeNoFollow URL||
 +|NonUniqueLinkTypeProtocolHTTPS RootDomain||
 +|NonUniqueLinkTypeProtocolHTTPS SubDomain||
 +|NonUniqueLinkTypeProtocolHTTPS URL||
 +|NonUniqueLinkTypeRedirect RootDomain||
 +|NonUniqueLinkTypeRedirect SubDomain||
 +|NonUniqueLinkTypeRedirect URL||
 +|NonUniqueLinkTypeTextLink RootDomain||
 +|NonUniqueLinkTypeTextLink SubDomain||
 +|NonUniqueLinkTypeTextLink URL||
 +|OutDomainsExternal RootDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.|
 +|OutDomainsExternal SubDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.|
 +|OutDomainsExternal URL|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksExternal RootDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksExternal SubDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksExternal URL|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksInternal RootDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksInternal SubDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksInternal URL|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.|
 +|OutLinksPages RootDomain||
 +|OutLinksPages SubDomain||
 +|OutLinksPages URL||
 +|RedirectFlag RootDomain|was it redirecting|
 +|RedirectFlag SubDomain|was it redirecting|
 +|RedirectFlag URL|was it redirecting|
 +|RedirectTo RootDomain|Final redirecting URL|
 +|RedirectTo SubDomain|Final redirecting URL|
 +|RedirectTo URL|Final redirecting URL|
 +|RefDomainsEDU RootDomain|Number of referring .EDU/.AC/.EDU.xx domains|
 +|RefDomainsEDU SubDomain|Number of referring .EDU/.AC/.EDU.xx domains|
 +|RefDomainsEDU URL|Number of referring .EDU/.AC/.EDU.xx domains|
 +|RefDomainsGOV RootDomain|Number of referring .GOV/.MIL/.GOV.xx domains.|
 +|RefDomainsGOV SubDomain|Number of referring .GOV/.MIL/.GOV.xx domains.|
 +|RefDomainsGOV URL|Number of referring .GOV/.MIL/.GOV.xx domains.|
 +|RefDomainTypeDirect RootDomain||
 +|RefDomainTypeDirect SubDomain||
 +|RefDomainTypeDirect URL||
 +|RefDomainTypeFollow RootDomain||
 +|RefDomainTypeFollow SubDomain||
 +|RefDomainTypeFollow URL||
 +|RefDomainTypeHomepageLink RootDomain||
 +|RefDomainTypeHomepageLink SubDomain||
 +|RefDomainTypeHomepageLink URL||
 +|RefDomainTypeLive RootDomain||
 +|RefDomainTypeLive SubDomain||
 +|RefDomainTypeLive URL||
 +|RefDomainTypeProtocolHTTPS RootDomain||
 +|RefDomainTypeProtocolHTTPS SubDomain||
 +|RefDomainTypeProtocolHTTPS URL||
 +|RefIPs RootDomain|Number of referring IP addresses pointing to this URL|
 +|RefIPs SubDomain|Number of referring IP addresses pointing to this URL|
 +|RefIPs URL|Number of referring IP addresses pointing to this URL|
 +|RefLanguage RootDomain||
 +|RefLanguage SubDomain||
 +|RefLanguage URL||
 +|RefLanguageConfidence RootDomain||
 +|RefLanguageConfidence SubDomain||
 +|RefLanguageConfidence URL||
 +|RefLanguageDesc RootDomain||
 +|RefLanguageDesc SubDomain||
 +|RefLanguageDesc URL||
 +|RefLanguagePageRatios RootDomain||
 +|RefLanguagePageRatios SubDomain||
 +|RefLanguagePageRatios URL||
 +|RefLanguageTotalPages RootDomain||
 +|RefLanguageTotalPages SubDomain||
 +|RefLanguageTotalPages URL||
 +|RefSubNets RootDomain|Number of referring IP Class C subnets pointing to this URL|
 +|RefSubNets SubDomain|Number of referring IP Class C subnets pointing to this URL|
 +|RefSubNets URL|Number of referring IP Class C subnets pointing to this URL|
 +|RootDomainIPAddress RootDomain||
 +|RootDomainIPAddress SubDomain||
 +|RootDomainIPAddress URL||
 +|TotalNonUniqueLinks RootDomain||
 +|TotalNonUniqueLinks SubDomain||
 +|TotalNonUniqueLinks URL||
 +|TrustMetric RootDomain||
 +|TrustMetric SubDomain||
 +|TrustMetric URL||
 +
 +===== DomDetailer (requires API-Key) =====
 +You can get an API-Key on the [[https://domdetailer.com/|DomDetailer Homepage]].
 +
 +^Column^Description^
 +|DD MozLinks|The number of links (equity or nonequity or not, internal or external) to the URL|
 +|DD MozPage Authority|A normalized 100-point score representing the likelihood of a page to rank well in search engine results|
 +|DD MozDomain Authority|A normalized 100-point score representing the likelihood of a domain to rank well in search engine results|
 +|DD MozRank|The MozRank of the URL, in the normalized 10-point score|
 +|DD MozTrust|The MozTrust of the URL, in the normalized 10-point|
 +|DD MozSpam|Bit field of triggered spam flags|
 +|DD Facebook Comments|Facebook Comments|
 +|DD Facebook Shares|Facebook Shares|
 +|DD Google+ Shares|Google+ shares|
 +|DD Pinterest Pins|pinterest shares|
 +|DD LinkedIn Shares|LinkedIn shares|
 +|DD majesticLinks|Number of backlinks to this URL|
 +|DD majesticRefDomains|Number of unique domains linking to this URL|
 +|DD majesticRefDomainsEDU|Number of unique .edu domains linking to this URL|
 +|DD majesticRefDomainsGOV|Number of unique .gov domains linking to this URL|
 +|DD majesticRefSubnets|Number of unique subnets linking to this URL|
 +|DD majesticTrustFlow|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors|
 +|DD majesticCitationFlow|Number of predicting how influential a URL might be based on how many sites link to it|
 +|DD majesticCat|Suggested Category of this site.|
 +
 +===== SEORank (needs API-Key) =====
 +You can get an API-Key on the [[https://seo-rank.my-addr.com/|SEORank Homepage]].
 +
 +^Column^Description^
 +|SEORank Domain Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how DOMAIN will be ranked on search engines, DA linked with link counts, MozRank, MozTrust scores and other data.|
 +|SEORank Page Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how URL will be ranked on search engines, PA linked with link counts, MozRank, MozTrust and other data.|
 +|SEORank Moz Rank|represents a link popularity score that showing the importance of given web page on the Internet.|
 +|SEORank Links In|number of links (equity or nonequity or not, internal or external) to the URL (data provided by Moz).|
 +|SEORank External Equity Links|number of external equity links to the URL (data provided by Moz).|
 +|SEORank Alexa Rank|global Alexa rank of domain.|
 +|SEORank ALexa Links number|number of links to domain.|
 +|SEORank Alexa Country|ISO2 code of country where domain is the most popular.|
 +|SEORank Alexa Country Rank|Alexa Rank of domain in country where it is popular.|
 +|SEORank SemRush Domain|domain that was taken by SemRush from url for analysis.|
 +|SEORank SemRush Rank|rank of domain according to SemRush.|
 +|SEORank SemRush Keywords number|number of keywords where site in Google|
 +|SEORank SemRush Traffic|number of users expected to visit the website during the following month.|
 +|SEORank SemRush Costs|estimated price of organic keywords in Google AdWords.|
 +|SEORank SemRush URL Links number|number of links to URL according to SemRush.|
 +|SEORank SemRush HOSTNAME Links number|number of links to HOSTNAME.|
 +|SEORank SemRush DOMAIN Links number|number of links to SemRush Domain .|
 +|SEORank Spam Score|number from 0 to 17, highest number means highest percent of sites that contains link to url but penalized or banned by Google.|
 +|SEORank Citation Flow|number from 0 to 100 (higher is better), it displaying how influential a URL might be based on how many sites link to it.|
 +|SEORank Trust Flow|number from 0 to 100 (higher is better), it displaying how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors.|
 +|SEORank ExtBackLinks|number of external backlinks to current URL (data provided by Majestic).|
 +|SEORank Refered Domains|number of domains with pages that refered to url|
  
 ====== Google-PR Emulation ====== ====== Google-PR Emulation ======