Hi everyone, I’ve been building my own log search server because I wasn’t satisfied with any of the alternatives out there and wanted a project to learn rust with. It still needs a ton of work but wanted to share what I’ve built so far.

The repo is up here: https://codeberg.org/Kryesh/crystalline

and i’ve started putting together some documentation here: https://kryesh.codeberg.page/crystalline/

There’s a lot of features I plan to add to it but I’m curious to hear what people think and if there’s anything you’d like to see out of a project like this.

Some examples from my lab environment:

events view searching for SSH logins from systemd journals and syslog events:

counting raw event size for all indices:

performance is looking pretty decent so far, and it can be configured to not be too much of a resource hog depending on use case, some numbers from my test install:

  • raw events ingested: ~52 million
  • raw event size: ~40GB
  • on disk size: ~5.8GB

Ram usage:

  • not running searches ingesting 600MB-1GB per day it uses about 500MB of ram
  • running the ssh search examples above brings it to about 600MB of ram while the search is running
  • running last example search getting the size of all events (requires decompressing the entire event store) peaked at about 3.5GB of ram usage
  • warmaster@lemmy.worldEnglish
    81·
    11 days ago

    That looks great, congrats!

    If you’re targeting us, homelabbers, I’ll tell you what I would want from a log server:

    • Stupid easy installation (Docker / Proxmox)
    • Integrations with:
    • Proxmox
    • Docker
    • Home Assistant
    • Frigate
    • Scrypted
    • Jellyfin
    • Immich
    • Unifi
    • Kryesh@lemmy.worldOPEnglish
      1·
      11 days ago

      Thanks! definitely aiming for a stupid easy installation/management for the app itself; but in my experience getting a wide range of supported log sources is no small feat. I’ve been using fluentbit to handle collection from different sources and using the following has been working well for me:

      • docker ‘journald’ log driver
      • fluentbit ‘systemd’ input
      • fluentbit ‘http’ output like the one in the readme

      with that setup you can search for container logs by name which works great with compose:

      or process logs from an nginx container like this to see traffic from external hosts:

      I’ll add a more complete example to the docs, but if you look in the repo there’s a complete example for receiving and ingesting syslog that you can run with just “docker compose up”