Give Your AI Agent Private Web Search: Self-Host SearXNG

Here's a problem nobody warns you about until you hit it: the moment your AI agent needs to look something up on the web, you're suddenly on the hook for a search API. Google's Custom Search caps you at 100 free queries a day and then charges per thousand. Bing's API pricing got worse, not better. And every query your agent runs, every half-formed research question, every internal document title it decides to Google, gets logged on someone else's servers with your API key attached.

For a hobby bot that's annoying. For an agent touching anything sensitive, it's a leak you signed up for on purpose.

SearXNG fixes both problems at once. It's a self-hosted metasearch engine: it takes your query, fans it out across dozens of real search engines (Google, Bing, DuckDuckGo, Wikipedia, GitHub, and so on), aggregates the results, and hands them back. No API key, no per-query billing, no query logs leaving your box. Point it at your own server and your agent gets a clean JSON search endpoint that costs you nothing beyond the VM it runs on.

Let me show you how to stand one up and, more importantly, the one config change that trips up almost everyone.

What you'll need

• A SearXNG instance (we'll deploy one on Elestio in a minute)
• An agent or app that can make HTTP requests (OpenWebUI, LiteLLM, n8n, LangChain, or your own code)
• Five minutes

Deploy SearXNG in a couple of clicks

You can run SearXNG from its Docker Compose stack yourself, but the default setup involves a Redis-backed limiter, a secret key you have to generate, and a reverse proxy you have to wire up. It's fiddly for something you just want working.

The managed route on Elestio skips all of that. You get an instance with SSL, automated backups, and updates handled, starting around $11/month on a small VM. Pick a provider and region, hit deploy, and you'll have an HTTPS endpoint in a few minutes. Whichever way you go, the rest of this guide is identical.

The part everyone gets wrong: turn on JSON

Here's the thing that cost me an embarrassing amount of time the first time. Out of the box, SearXNG only returns HTML. It's built for humans in a browser, not agents. If you point your code at it and ask for JSON, you get a flat 403 Forbidden and no explanation.

You have to explicitly enable the JSON format. Open your settings.yml and find the search block:

search:
  formats:
    - html
    - json

That - json line is the whole trick. Save it, restart SearXNG, and the API comes to life. On Elestio you edit this through the dashboard's config files and restart the service; if you're self-managing Compose, it lives in the volume you mounted for settings.yml.

While you're in there, make sure you've got a unique secret_key set under server. SearXNG won't start with the default placeholder, and a weak one makes the limiter easy to abuse.

Querying it

Now the fun part. A search is a plain GET request:

curl 'https://your-searxng.example.io/search?q=self-hosted+vector+database&format=json'

You get back a JSON object with a results array, plus suggestions, infoboxes, and number_of_results. Each result has the fields your agent actually cares about: title, url, content (the snippet), and engine.

A few parameters are worth knowing because they make agent results dramatically better:

The time_range=month one is quietly the most useful for agents. A RAG pipeline pulling stale results from three years ago is worse than useless, and one parameter fixes it.

Wiring it into your agent

Most agent tooling already speaks SearXNG, so you rarely have to write the HTTP call yourself:

• OpenWebUI: set the web search engine to searxng and paste your instance URL with <query> as the placeholder. Local chat gets live web results instantly.
• LiteLLM: it ships a SearXNG search integration, so your gateway can hand any model fresh results without a paid search key.
• n8n: there's a dedicated SearXNG tool node for LangChain agents. Drop it in, point it at your URL, done.
• Custom code: it's just a GET request. Parse results[].url and results[].content and feed them into your prompt.

Troubleshooting

Every request returns 403 Forbidden. You didn't enable the JSON format, or you didn't restart after editing settings.yml. Go back to that formats block. This is the number one issue by a wide margin.

Requests work in a browser but fail from your agent (429 or empty results). SearXNG has a bot-protection limiter that throttles clients it thinks are scraping, and an automated agent looks exactly like a scraper. For a private, single-user instance you can relax it in limiter.toml or disable it outright. Do not disable the limiter on a public instance, or you'll become someone's free proxy.

Results come back thin or empty. Some upstream engines rate-limit or temporarily block SearXNG. That's normal. Because it queries dozens of engines, one going quiet barely dents your results, but if many do, check the built-in /stats page to see which engines are erroring and disable the dead ones.

It ran fine, then stopped after a while. Usually a memory pinch on a tiny VM once you're hammering it with agent traffic. Bump the instance size a notch; SearXNG itself is light, but concurrent engine requests add up.

Why this is worth the ten minutes

Once it's running, you stop thinking about it. Your agents get unlimited web search, your queries stay on infrastructure you control, and you've deleted a recurring API bill from your stack. For anything doing research, RAG, or fact-checking, private search stops being a line item and becomes part of the furniture.

If you'd rather not babysit the Compose file, the managed SearXNG on Elestio gets you an SSL endpoint with backups and updates handled, and you can spend your time on the agent instead of the search plumbing.

Thanks for reading ❤️ See you in the next one 👋