A new tutorial has been released, allowing users to chat with any full website, not just a single page. This feature is achieved by crawling the entire website, extracting and cleaning the data, and then embedding it for vector search. The author of the tutorial emphasizes the importance of control over the crawler and extraction process to ensure reliable results.

The tutorial uses Apify for data extraction and Open WebUI for embedding and using the data for quality assurance (QA) with a Large Language Model (LLM). The author suggests breaking down the workflow into pieces, using specialized tools for each part, rather than relying on one vendor to do it all.

This innovative approach can be useful for individuals who need to analyze or extract information from complex websites. With this tutorial, users can ask questions about the entire website and receive answers in natural language.

Source: https://dev.to/airabbit/chat-with-any-full-website-not-just-a-single-page-complete-tutorial-294k