Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revisionLast revisionBoth sides next revision | ||
proxy_scraper:url_metrics [2017-02-13 22:37] – sven | proxy_scraper:url_metrics [2020-05-12 14:50] – [DomDetailer (requires API-Key)] sven | ||
---|---|---|---|
Line 13: | Line 13: | ||
After resolving the data, you can export or filter them. | After resolving the data, you can export or filter them. | ||
- | Please note that unlike other tools, **this module does not require any accounts** to get the data. All you need are proxies which the program | + | Here is a list of all possible metrics |
- | At startup it might be a bit slow as the software has to find proxies that would work. But once found they get cached and processing should be faster. Even after a program restart it should be faster. | + | ===== Metrics accessible without any API-Key ===== |
- | Here is a list of all possible metrics you can extract: | ||
^Column^Description^ | ^Column^Description^ | ||
- | |Title|The title of the page, if available| | ||
|Yandex TIC URL|Yandex TIC for URL| | |Yandex TIC URL|Yandex TIC for URL| | ||
|Yandex TIC SubDomain|Yandex TIC for SubDomain| | |Yandex TIC SubDomain|Yandex TIC for SubDomain| | ||
|Yandex TIC RootDomain|Yandex TIC for RootDomain| | |Yandex TIC RootDomain|Yandex TIC for RootDomain| | ||
- | |SEMRush Rank-SubDomain|SEMRush Rank for SubDomain| | + | |Yandex AGS SubDomain|Yandex TIC for SubDomain| |
- | |SEMRush Rank-RootDomain|SEMRush Rank for RootDomain| | + | |Yandex AGS RootDomain|Yandex TIC for RootDomain| |
- | |SEMRush Cost-SubDomain|how much need to spend for get same number of visitors from PPC for SubDomain| | + | |
- | |SEMRush Cost-RootDomain|how much need to spend for get same number of visitors from PPC for RootDomain| | + | |
- | |SEMRush Traffic-SubDomain|estimated number of visitors coming from search engines for SubDomain| | + | |
- | |SEMRush Traffic-RootDomain|estimated number of visitors coming from search engines for RootDomain| | + | |
- | |SEMRush Keywords-SubDomain|number of ranked keywords in google ranking for SubDomain| | + | |
- | |SEMRush Keywords-RootDomain|number of ranked keywords in google ranking | + | |
|Alexa Traffic Rank|Alexa Traffic Rank| | |Alexa Traffic Rank|Alexa Traffic Rank| | ||
|WebArchive Date|Oldest Webarchive Date| | |WebArchive Date|Oldest Webarchive Date| | ||
- | |Facebook Shares|Number of Facebook Shares| | ||
- | |Facebook Comments|Number of Facebook Comments| | ||
- | |Google+ Shares|Number of Google+ Shares| | ||
|URL Status|Informations about the URL| | |URL Status|Informations about the URL| | ||
|Google Indexed Pages|How many URLs are indexed for this SubDomain| | |Google Indexed Pages|How many URLs are indexed for this SubDomain| | ||
+ | |Google Indexed URL|Is this URLs indexed in Google?| | ||
+ | |Country|Country that this website is for| | ||
+ | |Language|Language that this website is in| | ||
+ | ===== Metrics accessible only via PR Emulation Software ===== | ||
+ | ^Column^Description^ | ||
+ | |PR URL|PageRank by Google for URL| | ||
+ | |PR SubDomain|PageRank by Google for SubDomain| | ||
+ | |PR RootDomain|PageRank by Google for RootDomain| | ||
+ | |||
+ | ===== Majestic (requires API-key) ===== | ||
+ | You can get an API-Key on the [[https:// | ||
+ | ^Column^Description^ | ||
+ | |TrustFlow URL|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
+ | |TrustFlow SubDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
+ | |TrustFlow RootDomain|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
+ | |CitationFlow URL|Number of predicting how influential a URL might be based on how many sites link to it| | ||
+ | |CitationFlow SubDomain|Number of predicting how influential a URL might be based on how many sites link to it| | ||
+ | |CitationFlow RootDomain|Number of predicting how influential a URL might be based on how many sites link to it| | ||
+ | |ExtBackLinks URL|Number of backlinks to this URL| | ||
+ | |ExtBackLinks SubDomain|Number of backlinks to this SubDomain| | ||
+ | |ExtBackLinks RootDomain|Number of backlinks to this RootDomain| | ||
+ | |RefDomains URL|Number of unique domains linking to this URL| | ||
+ | |RefDomains SubDomain|Number of unique domains linking to this SubDomain| | ||
+ | |RefDomains RootDomain|Number of unique domains linking to this RootDomain| | ||
+ | |ACRank RootDomain|CitationFlow like Rank (but less relevant)| | ||
+ | |ACRank SubDomain|CitationFlow like Rank (but less relevant)| | ||
+ | |ACRank URL|CitationFlow like Rank (but less relevant)| | ||
+ | |CrawledFlag RootDomain|Was it actually crawled| | ||
+ | |CrawledFlag SubDomain|Was it actually crawled| | ||
+ | |CrawledFlag URL|Was it actually crawled| | ||
+ | |CrawledURLs RootDomain|| | ||
+ | |CrawledURLs SubDomain|| | ||
+ | |CrawledURLs URL|| | ||
+ | |ExtBackLinksEDU RootDomain|Number of external .EDU/ | ||
+ | |ExtBackLinksEDU SubDomain|Number of external .EDU/ | ||
+ | |ExtBackLinksEDU URL|Number of external .EDU/ | ||
+ | |ExtBackLinksGOV RootDomain|Number of external .GOV/ | ||
+ | |ExtBackLinksGOV SubDomain|Number of external .GOV/ | ||
+ | |ExtBackLinksGOV URL|Number of external .GOV/ | ||
+ | |FinalRedirectResult RootDomain|Describes the response from attempting to crawl the page after all redirects are followed| | ||
+ | |FinalRedirectResult SubDomain|Describes the response from attempting to crawl the page after all redirects are followed| | ||
+ | |FinalRedirectResult URL|Describes the response from attempting to crawl the page after all redirects are followed| | ||
+ | |IndexedURLs RootDomain|Amount of indexted URLs| | ||
+ | |IndexedURLs SubDomain|Amount of indexted URLs| | ||
+ | |IndexedURLs URL|Amount of indexted URLs| | ||
+ | |Language RootDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
+ | |Language SubDomain|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
+ | |Language URL|For URLs, this is the language code for the source page. For subdomains and root domains, this is the detected languages for each page on that domain. This may be a comma delimited string to support multiple languages (i.e. en,de).| | ||
+ | |LanguageConfidence RootDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
+ | |LanguageConfidence SubDomain|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
+ | |LanguageConfidence URL|Percentages indicating the confidence of the language. This may be a comma delimited string to support multiple languages (i.e. 90,80).| | ||
+ | |LanguageDesc RootDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
+ | |LanguageDesc SubDomain|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
+ | |LanguageDesc URL|This is the English name of the language codes. This may be a comma delimited string to support multiple languages (i.e. English, | ||
+ | |LanguagePageRatios RootDomain|| | ||
+ | |LanguagePageRatios SubDomain|| | ||
+ | |LanguagePageRatios URL|| | ||
+ | |LanguageTotalPages RootDomain|| | ||
+ | |LanguageTotalPages SubDomain|| | ||
+ | |LanguageTotalPages URL|| | ||
+ | |LastCrawlDate RootDomain|Date when it was crawled| | ||
+ | |LastCrawlDate SubDomain|Date when it was crawled| | ||
+ | |LastCrawlDate URL|Date when it was crawled| | ||
+ | |LastCrawlResult RootDomain|Status code when being crawled| | ||
+ | |LastCrawlResult SubDomain|Status code when being crawled| | ||
+ | |LastCrawlResult URL|Status code when being crawled| | ||
+ | |LastSeen RootDomain|| | ||
+ | |LastSeen SubDomain|| | ||
+ | |LastSeen URL|| | ||
+ | |NonUniqueLinkTypeDeleted RootDomain|| | ||
+ | |NonUniqueLinkTypeDeleted SubDomain|| | ||
+ | |NonUniqueLinkTypeDeleted URL|| | ||
+ | |NonUniqueLinkTypeFrame RootDomain|| | ||
+ | |NonUniqueLinkTypeFrame SubDomain|| | ||
+ | |NonUniqueLinkTypeFrame URL|| | ||
+ | |NonUniqueLinkTypeHomepages RootDomain|| | ||
+ | |NonUniqueLinkTypeHomepages SubDomain|| | ||
+ | |NonUniqueLinkTypeHomepages URL|| | ||
+ | |NonUniqueLinkTypeImageLink RootDomain|| | ||
+ | |NonUniqueLinkTypeImageLink SubDomain|| | ||
+ | |NonUniqueLinkTypeImageLink URL|| | ||
+ | |NonUniqueLinkTypeIndirect RootDomain|| | ||
+ | |NonUniqueLinkTypeIndirect SubDomain|| | ||
+ | |NonUniqueLinkTypeIndirect URL|| | ||
+ | |NonUniqueLinkTypeNoFollow RootDomain|| | ||
+ | |NonUniqueLinkTypeNoFollow SubDomain|| | ||
+ | |NonUniqueLinkTypeNoFollow URL|| | ||
+ | |NonUniqueLinkTypeProtocolHTTPS RootDomain|| | ||
+ | |NonUniqueLinkTypeProtocolHTTPS SubDomain|| | ||
+ | |NonUniqueLinkTypeProtocolHTTPS URL|| | ||
+ | |NonUniqueLinkTypeRedirect RootDomain|| | ||
+ | |NonUniqueLinkTypeRedirect SubDomain|| | ||
+ | |NonUniqueLinkTypeRedirect URL|| | ||
+ | |NonUniqueLinkTypeTextLink RootDomain|| | ||
+ | |NonUniqueLinkTypeTextLink SubDomain|| | ||
+ | |NonUniqueLinkTypeTextLink URL|| | ||
+ | |OutDomainsExternal RootDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutDomainsExternal SubDomain|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutDomainsExternal URL|For URLs, this is the number of outgoing links to unique domains for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksExternal RootDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksExternal SubDomain|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksExternal URL|For URLs, this is the number of outgoing links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksInternal RootDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksInternal SubDomain|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksInternal URL|For URLs, this is the number of internal links for that page. For subdomains and root domains, average numbers are shown.| | ||
+ | |OutLinksPages RootDomain|| | ||
+ | |OutLinksPages SubDomain|| | ||
+ | |OutLinksPages URL|| | ||
+ | |RedirectFlag RootDomain|was it redirecting| | ||
+ | |RedirectFlag SubDomain|was it redirecting| | ||
+ | |RedirectFlag URL|was it redirecting| | ||
+ | |RedirectTo RootDomain|Final redirecting URL| | ||
+ | |RedirectTo SubDomain|Final redirecting URL| | ||
+ | |RedirectTo URL|Final redirecting URL| | ||
+ | |RefDomainsEDU RootDomain|Number of referring .EDU/ | ||
+ | |RefDomainsEDU SubDomain|Number of referring .EDU/ | ||
+ | |RefDomainsEDU URL|Number of referring .EDU/ | ||
+ | |RefDomainsGOV RootDomain|Number of referring .GOV/ | ||
+ | |RefDomainsGOV SubDomain|Number of referring .GOV/ | ||
+ | |RefDomainsGOV URL|Number of referring .GOV/ | ||
+ | |RefDomainTypeDirect RootDomain|| | ||
+ | |RefDomainTypeDirect SubDomain|| | ||
+ | |RefDomainTypeDirect URL|| | ||
+ | |RefDomainTypeFollow RootDomain|| | ||
+ | |RefDomainTypeFollow SubDomain|| | ||
+ | |RefDomainTypeFollow URL|| | ||
+ | |RefDomainTypeHomepageLink RootDomain|| | ||
+ | |RefDomainTypeHomepageLink SubDomain|| | ||
+ | |RefDomainTypeHomepageLink URL|| | ||
+ | |RefDomainTypeLive RootDomain|| | ||
+ | |RefDomainTypeLive SubDomain|| | ||
+ | |RefDomainTypeLive URL|| | ||
+ | |RefDomainTypeProtocolHTTPS RootDomain|| | ||
+ | |RefDomainTypeProtocolHTTPS SubDomain|| | ||
+ | |RefDomainTypeProtocolHTTPS URL|| | ||
+ | |RefIPs RootDomain|Number of referring IP addresses pointing to this URL| | ||
+ | |RefIPs SubDomain|Number of referring IP addresses pointing to this URL| | ||
+ | |RefIPs URL|Number of referring IP addresses pointing to this URL| | ||
+ | |RefLanguage RootDomain|| | ||
+ | |RefLanguage SubDomain|| | ||
+ | |RefLanguage URL|| | ||
+ | |RefLanguageConfidence RootDomain|| | ||
+ | |RefLanguageConfidence SubDomain|| | ||
+ | |RefLanguageConfidence URL|| | ||
+ | |RefLanguageDesc RootDomain|| | ||
+ | |RefLanguageDesc SubDomain|| | ||
+ | |RefLanguageDesc URL|| | ||
+ | |RefLanguagePageRatios RootDomain|| | ||
+ | |RefLanguagePageRatios SubDomain|| | ||
+ | |RefLanguagePageRatios URL|| | ||
+ | |RefLanguageTotalPages RootDomain|| | ||
+ | |RefLanguageTotalPages SubDomain|| | ||
+ | |RefLanguageTotalPages URL|| | ||
+ | |RefSubNets RootDomain|Number of referring IP Class C subnets pointing to this URL| | ||
+ | |RefSubNets SubDomain|Number of referring IP Class C subnets pointing to this URL| | ||
+ | |RefSubNets URL|Number of referring IP Class C subnets pointing to this URL| | ||
+ | |RootDomainIPAddress RootDomain|| | ||
+ | |RootDomainIPAddress SubDomain|| | ||
+ | |RootDomainIPAddress URL|| | ||
+ | |TotalNonUniqueLinks RootDomain|| | ||
+ | |TotalNonUniqueLinks SubDomain|| | ||
+ | |TotalNonUniqueLinks URL|| | ||
+ | |TrustMetric RootDomain|| | ||
+ | |TrustMetric SubDomain|| | ||
+ | |TrustMetric URL|| | ||
+ | |||
+ | ===== DomDetailer (requires API-Key) ===== | ||
+ | You can get an API-Key on the [[https:// | ||
+ | |||
+ | ^Column^Description^ | ||
+ | |DD MozLinks|The number of links (equity or nonequity or not, internal or external) to the URL| | ||
+ | |DD MozPage Authority|A normalized 100-point score representing the likelihood of a page to rank well in search engine results| | ||
+ | |DD MozDomain Authority|A normalized 100-point score representing the likelihood of a domain to rank well in search engine results| | ||
+ | |DD MozRank|The MozRank of the URL, in the normalized 10-point score| | ||
+ | |DD MozTrust|The MozTrust of the URL, in the normalized 10-point| | ||
+ | |DD MozSpam|Bit field of triggered spam flags| | ||
+ | |DD Facebook Comments|Facebook Comments| | ||
+ | |DD Facebook Shares|Facebook Shares| | ||
+ | |DD Google+ Shares|Google+ shares| | ||
+ | |DD Pinterest Pins|pinterest shares| | ||
+ | |DD LinkedIn Shares|LinkedIn shares| | ||
+ | |DD majesticLinks|Number of backlinks to this URL| | ||
+ | |DD majesticRefDomains|Number of unique domains linking to this URL| | ||
+ | |DD majesticRefDomainsEDU|Number of unique .edu domains linking to this URL| | ||
+ | |DD majesticRefDomainsGOV|Number of unique .gov domains linking to this URL| | ||
+ | |DD majesticRefSubnets|Number of unique subnets linking to this URL| | ||
+ | |DD majesticTrustFlow|Number predicting how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors| | ||
+ | |DD majesticCitationFlow|Number of predicting how influential a URL might be based on how many sites link to it| | ||
+ | |DD majesticCat|Suggested Category of this site.| | ||
+ | |||
+ | ===== SEORank (needs API-Key) ===== | ||
+ | |||
+ | ^Column^Description^ | ||
+ | |SEORank Domain Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how DOMAIN will be ranked on search engines, DA linked with link counts, MozRank, MozTrust scores and other data.| | ||
+ | |SEORank Page Authority|number from 1 to 100 (higher - better) provided by Moz and predicts how URL will be ranked on search engines, PA linked with link counts, MozRank, MozTrust and other data.| | ||
+ | |SEORank Moz Rank|represents a link popularity score that showing the importance of given web page on the Internet.| | ||
+ | |SEORank Links In|number of links (equity or nonequity or not, internal or external) to the URL (data provided by Moz).| | ||
+ | |SEORank External Equity Links|number of external equity links to the URL (data provided by Moz).| | ||
+ | |SEORank Alexa Rank|global Alexa rank of domain.| | ||
+ | |SEORank ALexa Links number|number of links to domain.| | ||
+ | |SEORank Alexa Country|ISO2 code of country where domain is the most popular.| | ||
+ | |SEORank Alexa Country Rank|Alexa Rank of domain in country where it is popular.| | ||
+ | |SEORank SemRush Domain|domain that was taken by SemRush from url for analysis.| | ||
+ | |SEORank SemRush Rank|rank of domain according to SemRush.| | ||
+ | |SEORank SemRush Keywords number|number of keywords where site in Google| | ||
+ | |SEORank SemRush Traffic|number of users expected to visit the website during the following month.| | ||
+ | |SEORank SemRush Costs|estimated price of organic keywords in Google AdWords.| | ||
+ | |SEORank SemRush URL Links number|number of links to URL according to SemRush.| | ||
+ | |SEORank SemRush HOSTNAME Links number|number of links to HOSTNAME.| | ||
+ | |SEORank SemRush DOMAIN Links number|number of links to SemRush Domain .| | ||
+ | |SEORank Spam Score|number from 0 to 17, highest number means highest percent of sites that contains link to url but penalized or banned by Google.| | ||
+ | |SEORank Citation Flow|number from 0 to 100 (higher is better), it displaying how influential a URL might be based on how many sites link to it.| | ||
+ | |SEORank Trust Flow|number from 0 to 100 (higher is better), it displaying how trustworthy a page is based on how trustworthy sites tend to link to trustworthy neighbors.| | ||
+ | |SEORank ExtBackLinks|number of external backlinks to current URL (data provided by Majestic).| | ||
+ | |SEORank Refered Domains|number of domains with pages that refered to url| | ||
====== Google-PR Emulation ====== | ====== Google-PR Emulation ====== |