User Tools

Site Tools


Sidebar

email_spider:add_new_search_engine

Add a new Search Engine

Let's say you want to add a new search engine like ebay. Than you have to perform the following Steps:

1. Finding the Right URL to Use

  • Open that URL in your browser and do a search of your choice that actually delivers many results

  • Right click on the page 2 link and copy that URL

You should now have an URL like this one:
http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=software&_pgn=2&_skc=50&rt=nc

The highlighted URL parts the the once that have to be modified before you add it to the program.

  • software must be exchanged with the place holder %search%
  • 2 is the page number and must be exchanged with the place holder %page%
    You can recognize this parameter by having a look at page 2,3,4… and see what numbered parameter increases. Usually it increases by 1 or by the numbe rof links you see on the page.

You new URL will look like this now:
http://www.ebay.com/sch/i.html?_from=R40&_sacat=0&_nkw=%search%&_pgn=%page%&_skc=50&rt=nc

Basically you have two types of search engines.

These will give you results of external links not belonging to the same domain as the search engine (e.g. google.com). In that case you copy all Links that have nothing to do with the results and make a big mask out of it.

Copy just something from the domains that are irrelevant like:
atdmt.com|google.|googleusercontent.|googleusercontent.|youtube.|blogger.com|schema.org

These will give you internal URLs that will point to sublinks on the same domain (as in this example). Let's copy like 3 URLs from the wanted result links and see what they have in common.

  1. http://www.ebay.com/itm/Microsoft-Office-2010-Home-and-Student-Retail-Box-3-PCs-NEW-/290923163259?pt=US_Office_Business_Software&hash=item43bc5f2e7b
  2. http://www.ebay.com/itm/Microsoft-Windows-7-Home-Premium-32-64-Bit-Upgrade-2-DVDs-/261269844948?pt=US_Operating_Systems_Software&hash=item3cd4e573d4
  3. http://www.ebay.com/itm/Microsoft-Office-2010-Home-and-Student-Product-Key-Card-PKC-Retail-79G-02020-NEW-/290923160299?pt=US_Office_Business_Software&hash=item43bc5f22eb

Exchange everything they have not in common with a “*”.

Now you should have an URL like this one:
http://www.ebay.com/itm/*/*?pt=*&hash=item*

3. Putting it all together

email_spider/add_new_search_engine.txt · Last modified: 2015-07-15 13:28 (external edit)