Quickwit + S3: Sub-Second Log Search Without Elasticsearch

I got a cloud bill once where the log storage line was bigger than the compute line. Not the app servers, not the database. The logs. We were paying premium block-storage prices to keep three replicas of data nobody looked at until something broke. If you have ever run Elasticsearch at scale, you know the exact number I am talking about, and you probably flinched just now.

Quickwit is the tool I wish I had back then. It is an open-source search engine built for logs and traces, and its whole design starts from one decision: keep the index on object storage (S3, GCS, Azure Blob) instead of on expensive local disks. That one choice is why the cost math looks so different.

The trick: search that runs off S3

Most search engines assume the index lives on fast local storage, right next to the CPU doing the searching. That is why Elasticsearch wants beefy nodes with lots of SSD, and why you replicate everything two or three times for safety. Storage and compute are welded together, so scaling one drags the other along.

Quickwit pulls them apart. Your index is stored as immutable files called "splits" that live directly in an S3 bucket. Searchers are stateless: they fetch only the tiny slices of a split they need to answer your query, cache the hot bits, and drop the rest. You can shut every searcher down overnight and your data sits safely in S3, costing you object-storage pennies instead of provisioned-SSD dollars.

The engine underneath is Tantivy, a Rust search library in the same family as Lucene. It is quick, and because the whole thing is written in Rust there is no JVM heap to babysit, which anyone who has tuned Elasticsearch garbage collection will appreciate.

What is actually running

A Quickwit deployment has a handful of parts, and you can run them all as a single binary or split them out as you grow:

• Indexer: ingests data and writes splits to object storage.
• Searcher: answers queries, stateless, scales horizontally.
• Metastore: tracks which splits exist and where. Use PostgreSQL in production; a file-based metastore on S3 is fine for smaller setups.
• Control plane: hands out indexing work.
• Janitor: runs retention and GDPR-style delete tasks so old logs actually disappear.

Data gets in through a REST ingest API, or from Kafka, Kinesis, and Pulsar if you already have a streaming pipeline. And here is the part observability folks care about: Quickwit is OpenTelemetry-native. It exposes OTLP endpoints for logs and traces directly, and it can act as a Jaeger backend, so your existing OTEL collectors and Jaeger UI just point at it. Searching goes through an Elasticsearch-compatible API, so a lot of existing clients and dashboards work without a rewrite, and there is an official Grafana data source.

One honest thing you should know

Quickwit joined Datadog in early 2025. Before you close the tab: the team relicensed the project under Apache 2.0 on the way in, and the repository is still getting commits in mid-2026. So it is genuinely open source, and you can self-host it today without asking anyone. Just go in with clear eyes. The core team's day job is now a Datadog product, so the OSS project leans on its community more than it did in 2023. For an append-only log and trace store, that is a trade I would still make, but you should make it knowingly.

Getting it running on Elestio

You can run Quickwit from a single Docker image, but you still have to wire up object storage, a Postgres metastore, TLS, and backups yourself. Elestio handles that part: managed Quickwit on a dedicated VM starts at $16/month, with SSL, automated backups, updates, and monitoring already configured. Deploy it from the Quickwit service page, pick a provider and region, and you have an endpoint in a few minutes.

Once it is up, define an index, then push logs into it. Quickwit needs the index to exist first, so create it once with a field mapping:

# Define an index called "logs" with three fields
curl -X POST "https://your-quickwit.elestio.app/api/v1/indexes" \
  -H "Content-Type: application/yaml" \
  --data-binary '
version: 0.8
index_id: logs
doc_mapping:
  timestamp_field: timestamp
  field_mappings:
    - name: timestamp
      type: datetime
      input_formats: [unix_timestamp]
      fast: true
    - name: severity
      type: text
    - name: message
      type: text
'

Now send it some data:

# Send a couple of log lines to the "logs" index
curl -X POST "https://your-quickwit.elestio.app/api/v1/logs/ingest" \
  -H "Content-Type: application/x-ndjson" \
  -d '{"timestamp": 1721460000, "severity": "ERROR", "message": "payment webhook timed out"}
{"timestamp": 1721460005, "severity": "INFO", "message": "retry succeeded"}'

Then search it with the native query API:

curl "https://your-quickwit.elestio.app/api/v1/logs/search?query=severity:ERROR"

If you are wiring up tracing, point your OpenTelemetry collector's OTLP exporter at the same host and add Quickwit as a Jaeger data source in Grafana. No schema migration, no reindex.

Quickwit vs Elasticsearch, honestly

Troubleshooting

• The first query feels slow: cold S3 reads are the price of decoupled storage. The hotcache warms after the first hit, so keep searchers running rather than scaling to zero if latency matters, and put them in the same region as your bucket.
• Ingest returns 200 but nothing shows up: splits are published on a commit interval, not instantly. Give it a few seconds and check the indexer logs for the commit. Data is near-real-time, not real-time.
• Metastore errors after adding a second indexer: a file-based metastore on S3 does not handle concurrent writers well. Move to PostgreSQL before you scale indexing out.
• Queries against S3 time out: double-check the bucket credentials and region in your environment. A surprising number of "Quickwit is broken" reports turn out to be an IAM policy missing s3:GetObject.

Quickwit will not replace a full-text product-search cluster, and it does not do document updates. But for logs and traces, which are append-only and enormous, running search straight off S3 is one of those ideas that feels obvious in hindsight.

Spin one up, point your logs at it, and watch what the storage line does on your next bill.

Thanks for reading ❤️ See you in the next one 👋