Building Your Own Metasearch Engine: A Deep Dive into Aggregation, Scraping, and Ranking

Before delving into the technical aspects, it's crucial to understand existing metasearch engines. Searx (and its successor, SearxNG) is a prominent open-source example, written in Python, supporting a vast array of search engines. It served as a significant inspiration for the author's project, particularly in its result display and ranking algorithm. However, Searx is noted for its slowness and perceived lack of hackability. Another well-known metasearch engine is Kagi, which sources results from its own crawler, Google, Yandex, Mojeek, Marginalia Search, and Brave. Kagi's unique feature is user-controlled domain ranking adjustments. Despite its appeal, Kagi's paid subscription model ($10/month) and limited customization options are deterrents for some users, including the author. Older metasearch engines like Dogpile and Metacrawler still exist but are considered less relevant for discussion.

The author's metasearch engine primarily relies on scraping Google, Bing, Brave, and Marginalia, eschewing APIs due to cost, result quality, and complexity. Web scraping involves parsing HTML content to extract desired information. For the Node.js implementation, Cheerio was used, while the Rust version employs the Scraper library. Both are effective tools for HTML parsing. The most challenging aspect of scraping is identifying the correct CSS selectors to pinpoint specific elements like search result containers, titles, links, and descriptions. For instance, Google's search result container might be identified by `div.g > div` or `div.xpd > div:first-child`, with titles often found in `h3` tags. Links are typically in `a[href]` attributes, though some engines require extracting text or decoding URL parameters. Extracting descriptions can be inconsistent, necessitating multiple selectors like `div[data-sncf]` and `div[style='-webkit-line-clamp:2']`. A key strategy is to use stable selectors (like `g` and `xpd` for Google) that are less prone to change during website updates, rather than randomly generated class names.

The ranking algorithm employed in 'metasearch2' is largely based on Searx's approach, proving surprisingly effective for its simplicity. The core idea is to score each search result based on its position across different aggregated engines and the weight assigned to each engine. The formula involves summing the product of an engine's weight and the inverse of the result's position within that engine. The author notes a slight deviation from Searx's original algorithm by omitting the multiplication by the number of occurrences, a change that did not negatively impact rankings. To ensure accurate merging of results from different engines, URL normalization is recommended. This includes converting URLs to HTTPS and removing trailing slashes, standardizing them for consistent comparison and aggregation.

Rendering search results is generally straightforward, as most search engines present information without overly complex styling. The choice of web or templating framework is flexible, but one that supports chunked responses is ideal. This allows for immediate delivery of the header HTML, followed by the search results, creating a perception of faster loading times for the user. For 'metasearch2', the author opted against a templating framework to maintain simplicity, instead building HTML manually. This approach also facilitated easy implementation of chunking and real-time progress updates for ongoing search engine requests, further enhancing the user experience.