Cribl Edge & Search Homelab Part 1 - Setup Minio

In my last post, I talked about how I’d be blogging a lot about my lessons from startup life. I’ve obviously failed at that over the last year, but now I’m back to describe how I’m running two of Cribl’s latest products in my homelab. There really is no practical purpose to doing this, as it would be more reliable and easier to forward this data to an actual AWS S3 bucket. But, it wouldn’t be as fun!

If you’d like to skip this step and use an actual AWS S3 bucket, setting that up is outside the scope of this article, but the rest of the steps should largely apply minus the Minio specifics (setting an endpoint in the Dataset Provider for example). Skip ahead if you like.

First step, we’re going to need a place to send the data. We’re going to use Minio. Minio is a fantastic project that provides a AWS S3 compatible API while storing the data locally. In my homelab, I have my desktop PC running Windows 10 that I use for hosting a number of Virtual Machines in Hyper-V. There are lots of ways to host VMs, choose your preferred method in your homelab. I chose to deploy on a VM because I wanted reliable storage and I didn’t want to deal with the complexities of high availability, containers, etc. Of course, if you want to be adventurous, you could run this workload in Kubernetes or in Docker or however you see fit. These instructions assume a VM. For this VM, I gave it 4 cores and 16 Gigs of RAM with a 30GB HD.

I’m starting with a freshly installed Ubuntu 22.04 VM, which I named minio. To setup Minio, I borrowed heavily from this gist. Copy and paste each line individually, making sure to change YOURADMINUSER, YOURADMINPASS, YOURACCESSKEY and YOURSECRETKEY to your preferences. In summary, what we’re doing is:

  1. Grabbing Minio bits
  2. Adding a user
  3. Installing minio in /usr/local/bin and setting permissions
  4. Setting up the Minio config in /etc/default/minio
  5. Creating a directory for Minio to store data and setting permissions
  6. Grabbing the systemd unit file and setting Minio up as a service and starting it
  7. Grabbing the Minio CLI mc, setting it up, and using it to add a access key/secret key and a bucket
cd ~
wget https://dl.min.io/server/minio/release/linux-amd64/minio

sudo useradd --system minio --shell /sbin/nologin
sudo usermod -L minio
sudo chage -E0 minio

sudo mv minio /usr/local/bin
sudo chmod +x /usr/local/bin/minio
sudo chown minio:minio /usr/local/bin/minio

sudo touch /etc/default/minio
echo 'MINIO_VOLUMES="/usr/local/share/minio"' | sudo tee -a /etc/default/minio
echo 'MINIO_OPTS="-C /etc/minio --address :9000"' | sudo tee -a /etc/default/minio
echo 'MINIO_ROOT_USER=clint' | sudo tee -a /etc/default/minio
echo 'MINIO_ROOT_PASSWORD=fuckfuckityfuck0' | sudo tee -a /etc/default/minio

sudo mkdir /usr/local/share/minio
sudo mkdir /etc/minio
sudo chown minio:minio /usr/local/share/minio
sudo chown minio:minio /etc/minio

cd ~

wget https://raw.githubusercontent.com/minio/minio-service/master/linux-systemd/minio.service

sed -i 's/User=minio-user/User=minio/g' minio.service
sed -i 's/Group=minio-user/Group=minio/g' minio.service

sudo mv minio.service /etc/systemd/system

sudo systemctl daemon-reload
sudo systemctl enable minio
sudo systemctl start minio

sudo systemctl status minio

wget https://dl.min.io/client/mc/release/linux-amd64/mc
sudo mv mc /usr/local/bin
sudo chmod +x /usr/local/bin/mc
sudo chown minio:minio /usr/local/bin/mc

mc alias set minio http://localhost:9000 YOURADMINUSER YOURADMINPASS
mc admin user add minio YOURACCESSKEY YOURSECRETKEY
mc admin policy set minio readwrite user=YOURACCESSKEY
mc mb minio/data

Minio should now be running, and you should be able to access it and login at http://YOURHOST:46517/. You should see a bucket in there called data.

Next, let’s get data flowing to Minio.

Posts in this Series