Differences
This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Next revisionBoth sides next revision | ||
proxy_scraper:options [2015-06-10 21:59] – [Provider] devin | proxy_scraper:options [2015-07-14 13:07] – [Provider] sven | ||
---|---|---|---|
Line 4: | Line 4: | ||
===== Settings ===== | ===== Settings ===== | ||
+ | {{ : | ||
==== Internal Proxy Server ==== | ==== Internal Proxy Server ==== | ||
Line 31: | Line 32: | ||
===== Provider ===== | ===== Provider ===== | ||
+ | {{ : | ||
A website that offers free proxy servers is called a // | A website that offers free proxy servers is called a // | ||
Line 41: | Line 43: | ||
When having this option checked, it will parse all those links for new proxies. | When having this option checked, it will parse all those links for new proxies. | ||
Often new proxy sources are found that way that you can later add here as well. In the proxy list it is shown as "// | Often new proxy sources are found that way that you can later add here as well. In the proxy list it is shown as "// | ||
+ | |||
+ | Another option called "**Use Search Engines to locate proxy lists**" | ||
Line 52: | Line 56: | ||
===== Automatic Export ===== | ===== Automatic Export ===== | ||
+ | {{ :export.png |}} | ||
There are many options for you to export proxies automatically. You can define an interval and different ways to export. | There are many options for you to export proxies automatically. You can define an interval and different ways to export. | ||
Line 60: | Line 65: | ||
* **WEB Upload** - This sends proxies to a webserver. You can define everything on how this should happen (POST/GET) | * **WEB Upload** - This sends proxies to a webserver. You can define everything on how this should happen (POST/GET) | ||
- | You can of course Edit or Delete | + | You can of course Edit or Delete |
- | Each Export offers | + | Each Export offers |
===== Automatic Search ===== | ===== Automatic Search ===== | ||
- | Having this checked means the program will search all the providers for new proxies. You can define the **Interval** here as well as the conditions on when this should happen or stopped. | + | {{ : |
+ | |||
+ | Having this checked means the program will search all the providers for new proxies. You can define the **Interval** here as well as the conditions on when this should happen or stop. | ||
The box with the different Tests is in fact very important. Usually people need proxies to be anonymous, you you should pick some tests here that offer this. | The box with the different Tests is in fact very important. Usually people need proxies to be anonymous, you you should pick some tests here that offer this. | ||
Also the //Google Search// test might be of interest for you. Other tests are also available like // | Also the //Google Search// test might be of interest for you. Other tests are also available like // | ||
- | Of course you can add your own tests here clicking the **Add** button or **Delete** | + | Of course you can add your own tests here clicking the **Add** button or **Delete** |
Each found proxy is tested against the selected tests. | Each found proxy is tested against the selected tests. | ||
Line 80: | Line 87: | ||
Anyway it is not suitable to keep dead proxies forever in the hope that they will work once again. You should not disable the option **Automatically remove proxies** in order to not wast memory and other resources. | Anyway it is not suitable to keep dead proxies forever in the hope that they will work once again. You should not disable the option **Automatically remove proxies** in order to not wast memory and other resources. | ||
- | The option to store removed proxies in an file enables you to test them all again once you feel like doing it. Thats possible clicking **Add -> Previously Removed** on the main interface. | + | The option to store removed proxies in an file enables you to test them all again if you want to go back and try that at a later time. That' |
===== Filter ===== | ===== Filter ===== | ||
- | I don't recommend you to use any of the filter options as you can always define filters for the automatic exports. However for those who want this, there is the possibility to do this. | + | {{ :filter.png |}} |
+ | |||
+ | I don't recommend you to use any of the filter options as you can always define filters for the automatic exports. However for those who want this, the following options are available: | ||
* **Do not accept anonymous (no elite) proxies**: This is not keeping proxies who identify them as such but not telling the remote website what IP you have. | * **Do not accept anonymous (no elite) proxies**: This is not keeping proxies who identify them as such but not telling the remote website what IP you have. | ||
* **Do not accept transparent proxies**: This is not keeping proxies who identify them as such AND tell the remote website what IP you have. | * **Do not accept transparent proxies**: This is not keeping proxies who identify them as such AND tell the remote website what IP you have. | ||
- | * **Skip suspicious proxies**: Suspicious proxies are those who probably spy on your activities on the proxy server itself. Keep in mind that many of the proxy servers you find are run by spy agencies. Anyway even if a proxy is not tagged as such, it can still spy on you. You can not trust anyone at all thees days. | + | * **Skip suspicious proxies**: Suspicious proxies are those who probably spy on your activities on the proxy server itself. Keep in mind that many of the proxy servers you find are run by spy agencies. Anyway even if a proxy is not tagged as such, it can still spy on you. You can not trust anyone at all these days. |
* **Accept only if tagged as**: Keep proxies only if they match a certain test and that test tagged the proxy as such. | * **Accept only if tagged as**: Keep proxies only if they match a certain test and that test tagged the proxy as such. | ||
- | * **Accept only the following ports** OR **Skip the following ports**: This can be usefull | + | * **Accept only the following ports** OR **Skip the following ports**: This can be useful |
- | * **Skip duplicate IPs**: Often you have proxies being on the same IP/Host but with different ports. This can be a problem if you got too many of those and do requests to search engines who only see the IP and check against that to see if you are hammering it. | + | * **Skip duplicate IPs**: Often you have proxies being on the same IP/Host but with different ports. This can be a problem if you have too many and do requests to search engines who only see the IP and check against that to see if you are hammering it. |
* **Accept only the following Types**: There are different types of proxies. WEB proxies are those that work like a normal http protocol but with a bit of a modification on the request header. They are only useful for web queries. Then you have Connect proxies, socks4 and socks5 who can basically be used for any purpose, not just website parsing. | * **Accept only the following Types**: There are different types of proxies. WEB proxies are those that work like a normal http protocol but with a bit of a modification on the request header. They are only useful for web queries. Then you have Connect proxies, socks4 and socks5 who can basically be used for any purpose, not just website parsing. | ||
- | * **Accept only the following Regions**: Proxies and there location on Earth are determinate | + | * **Accept only the following Regions**: Proxies and there location on Earth are determined |