running elasticsearch and kibana in a homelab - elk stack setup with docker
Elasticsearch + Kibana gives you a searchable log store and a SIEM (security information and event management) dashboard for your home network. here's how to set up the ELK stack in Docker on a homelab server, with security enabled from the start.
the elk stack components
- Elasticsearch - stores and indexes logs. the core of the ELK stack.
- Kibana - UI, search, dashboards, Elastic Security SIEM
- Filebeat - lightweight log shipper for Docker container logs
Logstash is skipped - Filebeat can write directly to Elasticsearch for a homelab this size.
why security enabled matters
ES 8.x ships with security enabled by default and it's worth keeping it on. without it, anyone on your network can read, modify, or delete your log data. with it, you get authentication and TLS between components.
the catch: enabling security on an existing ES volume that was initialized without it causes ES to fail on startup. the fix is to delete the data volume and let ES reinitialize. for a fresh setup, enable security from the start.
kibana can't use the elastic superuser
this tripped me up. Kibana 8 refuses to connect using the elastic superuser credentials — it logs an error and won't start. the fix is a service account token:
curl -X POST "http://localhost:9200/_security/service/elastic/kibana/credential/token/kibana_token" \
-u "elastic:yourpassword"
copy the value from the response and set it as ELASTICSEARCH_SERVICEACCOUNTTOKEN in the Kibana container environment. do not set ELASTICSEARCH_USERNAME and ELASTICSEARCH_PASSWORD — just the token.
three encryption keys
Elastic Security requires three separate encryption keys set as environment variables:
XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY
XPACK_SECURITY_ENCRYPTIONKEY
XPACK_REPORTING_ENCRYPTIONKEY
generate each one:
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 32
without all three, Kibana starts but the Security app throws privilege errors. once set, don't rotate them — rotating invalidates saved objects and dashboards.
memory map limit
ES requires a higher vm.max_map_count than the Linux default. set it on the host before starting:
sudo sysctl -w vm.max_map_count=262144
echo 'vm.max_map_count=262144' | sudo tee -a /etc/sysctl.conf
ES will fail to start without this.
filebeat as a separate stack
Filebeat runs as its own stack rather than in the same compose as ES/Kibana. the reasoning: Filebeat belongs on the host shipping logs, not bundled with the central store. eventually it'll run on multiple hosts, each shipping to the same ES instance.
the config file lives on the host at an absolute path and gets bind-mounted into the container. Portainer git stacks don't copy arbitrary files to remote agents — only the compose file gets deployed. so the config has to be placed on the host manually and mounted.
volumes:
- /opt/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
Filebeat reads container log files directly from the host path and enriches them with Docker metadata (container name, image, labels) via the Docker socket.
elasticsearch heap size and hardware requirements
ES defaults to auto-sizing heap, which can grab too much memory on a shared host. cap it explicitly:
ES_JAVA_OPTS=-Xms2g -Xmx2g
2GB is comfortable for a homelab with a few dozen containers shipping logs. for dedicated ELK stack hardware, a mini PC with 16-32GB RAM is the sweet spot - Elasticsearch is memory-hungry and benefits from fast storage for indexing. a fanless mini PC like a MinisForum keeps it quiet and runs 24/7 without issues.
what you get
once running, Kibana → Discover with a filebeat-* data view shows all Docker container logs with container name, image, and host metadata. you can filter to a specific container, search for errors, or set up alerts.
the Security app has hundreds of prebuilt detection rules for common attack patterns — port scans, brute force attempts, suspicious process execution. for a home network it's mostly noise, but it's useful for learning what normal traffic looks like and catching anything unexpected.
AMD Ryzen - compact, fanless, low power
Ideal for running Elasticsearch, Docker, and homelab services 24/7
view on amazonaffiliate link - helps support the blog
eli5
every program running on your servers writes a diary (logs). normally those diaries pile up in separate places and nobody reads them. Elasticsearch is a filing system that ingests all those diaries and makes them searchable in seconds. Kibana is the reading room where you browse and search them. Filebeat is the mail carrier that picks up the diaries from your containers and delivers them to the filing system. the "security enabled" part is like putting a lock on the filing room so not just anyone on your network can read or tamper with your logs. getting the lock set up right takes a few extra steps — service accounts, encryption keys — but it's worth doing from the start rather than retrofitting it later.