Cribl Edge & Search Homelab Part 1 - Setup Minio
In my last post, I talked about how I’d be blogging a lot about my lessons from startup life. I’ve obviously failed at that over the last year, but now I’m back to describe how I’m running two of Cribl’s latest products in my homelab. There really is no practical purpose to doing this, as it would be more reliable and easier to forward this data to an actual AWS S3 bucket. But, it wouldn’t be as fun!
If you’d like to skip this step and use an actual AWS S3 bucket, setting that up is outside the scope of this article, but the rest of the steps should largely apply minus the Minio specifics (setting an endpoint in the Dataset Provider for example). Skip ahead if you like.
First step, we’re going to need a place to send the data. We’re going to use Minio. Minio is a fantastic project that provides a AWS S3 compatible API while storing the data locally. In my homelab, I have my desktop PC running Windows 10 that I use for hosting a number of Virtual Machines in Hyper-V. There are lots of ways to host VMs, choose your preferred method in your homelab. I chose to deploy on a VM because I wanted reliable storage and I didn’t want to deal with the complexities of high availability, containers, etc. Of course, if you want to be adventurous, you could run this workload in Kubernetes or in Docker or however you see fit. These instructions assume a VM. For this VM, I gave it 4 cores and 16 Gigs of RAM with a 30GB HD.
I’m starting with a freshly installed Ubuntu 22.04 VM, which I named
minio. To setup Minio, I borrowed heavily from this gist. Copy and paste each line individually, making sure to change
YOURSECRETKEY to your preferences. In summary, what we’re doing is:
- Grabbing Minio bits
- Adding a user
- Installing minio in
/usr/local/binand setting permissions
- Setting up the Minio config in
- Creating a directory for Minio to store data and setting permissions
- Grabbing the systemd unit file and setting Minio up as a service and starting it
- Grabbing the Minio CLI
mc, setting it up, and using it to add a access key/secret key and a bucket
cd ~ wget https://dl.min.io/server/minio/release/linux-amd64/minio sudo useradd --system minio --shell /sbin/nologin sudo usermod -L minio sudo chage -E0 minio sudo mv minio /usr/local/bin sudo chmod +x /usr/local/bin/minio sudo chown minio:minio /usr/local/bin/minio sudo touch /etc/default/minio echo 'MINIO_VOLUMES="/usr/local/share/minio"' | sudo tee -a /etc/default/minio echo 'MINIO_OPTS="-C /etc/minio --address :9000"' | sudo tee -a /etc/default/minio echo 'MINIO_ROOT_USER=clint' | sudo tee -a /etc/default/minio echo 'MINIO_ROOT_PASSWORD=fuckfuckityfuck0' | sudo tee -a /etc/default/minio sudo mkdir /usr/local/share/minio sudo mkdir /etc/minio sudo chown minio:minio /usr/local/share/minio sudo chown minio:minio /etc/minio cd ~ wget https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service sed -i 's/User=minio-user/User=minio/g' minio.service sed -i 's/Group=minio-user/Group=minio/g' minio.service sudo mv minio.service /etc/systemd/system sudo systemctl daemon-reload sudo systemctl enable minio sudo systemctl start minio sudo systemctl status minio wget https://dl.min.io/client/mc/release/linux-amd64/mc sudo mv mc /usr/local/bin sudo chmod +x /usr/local/bin/mc sudo chown minio:minio /usr/local/bin/mc mc alias set minio http://localhost:9000 YOURADMINUSER YOURADMINPASS mc admin user add minio YOURACCESSKEY YOURSECRETKEY mc admin policy set minio readwrite user=YOURACCESSKEY mc mb minio/data
Minio should now be running, and you should be able to access it and login at http://YOURHOST:46517/. You should see a bucket in there called