matt - Network

Blue sky days out on the trail. #kitsuma #nc #mountainbike

Blue sky days out on the trail. #kitsuma #nc #mountainbike

Every time I go on vacation for a few days, some #homelab infra ends up breaking. It’s not a critical service, of course, which also means...

Every time I go on vacation for a few days, some #homelab infra ends up breaking. It’s not a critical service, of course, which also means I don’t have good remote access to fix it when it goes down. Need physical access.

At least I’ll be comforted by the constant ping of monitoring alerts telling me “hey, that box died”. Small comforts in a trying time.

https://rambo.codes/posts/2025-04-24-how-a-single-line-of-code-could-brick-your-iphone

https://rambo.codes/posts/2025-04-24-how-a-single-line-of-code-could-brick-your-iphone

Dedicated communities – on topics of photography, computers, Linux, cycling, etc – have the highest-minded debate and discussions of...

Dedicated communities – on topics of photography, computers, Linux, cycling, etc – have the highest-minded debate and discussions of issues.

A lot of topics I'm interested in Google tries to algorithmically feed me via their News app. The value of the content I consume there is low. Super low.

And I'm always reminded of that whenever I visit a website or chat room dedicated to a specific topic or passion.

9:35pm ET – welcome to the current time.I'm off with a screwdriver now to check the torque specs on "things”.And probably to...

9:35pm ET – welcome to the current time.

I'm off with a screwdriver now to check the torque specs on "things”.

And probably to buy a stand-off that will allow me to mount this drive to the motherboard without this pointless heat sink.

10/10

#proxmox #selfhosted #homelab

9:27pm ET – So...it just…came loose? The connection on the motherboard is falling apart? A screw was not tight? That "looks cool...

9:27pm ET – So...it just…came loose? The connection on the motherboard is falling apart? A screw was not tight? That "looks cool but does little" heat shield cause some sort of issue?

This machine has been running 24/7 in my house for about ~11 months since I built it brand new.

This is not the sort of failure I was expecting to see.

9/

#proxmox #selfhosted #homelab

9:23pm ET – Take it apart, put it back together.Nothing looks broken. It's not hot to the touch. It’s a PITA trying to assemble...

9:23pm ET – Take it apart, put it back together.

Nothing looks broken. It's not hot to the touch. It’s a PITA trying to assemble this thing sideways on my desk, though, and I keep dropping those tiny little screws in annoying places.

Power back on...no errors.

`zpool status`

It’s…there. wtf?

8/

#proxmox #selfhosted #homelab

9:19pm ET – Ok, the drive is physically there in it's slot.This consumer-grade mobo has one of those silly solid aluminum heat shields...

9:19pm ET – Ok, the drive is physically there in it's slot.

This consumer-grade mobo has one of those silly solid aluminum heat shields on top of it and the stand-offs to anchor the SSD down are shaped to specifically fit with this heat shield.

The shield has an adhesive back and sticks to the NVME drive (2280 size).

In the words of all the IT greats which came before me “have you tried plugging it in again?”

Machine now powered down, I remove the SSD.

7/

#proxmox #selfhosted #homelab

9:15pm ET – Back online.`zpool status` shows me the pool which has those VM images has NOT been mounted.`zpool import` does...nothing....

9:15pm ET – Back online.

`zpool status` shows me the pool which has those VM images has NOT been mounted.

`zpool import` does...nothing. 😬

Let's see if the disk is even recognized by the system.

`lsblk -f`
`fdisk -l`

Ok…that SSD drive is just NOT THERE ANYMORE.

Time to pop open the machine. Maybe, it, uh, fell out?

6/

#proxmox #selfhosted #homelab

9:12pm ET – we're rebooted but...no network connection. Bugger.Switch over to the physically connected keyboard and check out `ip a`...

9:12pm ET – we're rebooted but...no network connection. Bugger.

Switch over to the physically connected keyboard and check out `ip a` and then `/etc/network/interfaces`.

Yup, the port name changed from `enp3s0` to `enp2s0` and now my network config is broken. Ok, easy fix, just rename to match the current interface of `enp2s0`.

Reboot.

5/

#proxmox #selfhosted #homelab

9:10pm ET – Reboot (mercifully) finishes. Lots of errors dotting the log.Let's try booting back up, and see what the damage is.Good...

9:10pm ET – Reboot (mercifully) finishes. Lots of errors dotting the log.

Let's try booting back up, and see what the damage is.

Good news is that the boot disk is a separate zpool, so I should at least be able to get back into the proxmox environment and check on the zpool which hosts the VM data.

4/

#proxmox #selfhosted #homelab

9:06pm ET – some services have stopped, but the one ZFS pool on this box is VERY unhappy. It will not unmount.Seeing more...

9:06pm ET – some services have stopped, but the one ZFS pool on this box is VERY unhappy. It will not unmount.

Seeing more “uncorrectable I/O errors" piling up, too.

This ZFS pool hosts all my VM images when running. It's a single 2TB NVME SSD connected via a M.2 slot on a consumer-grade mobo.

VMs will not shut down or stop. Reboot keeps trying.

Looks like something bad has happened with that ZFS pool.

3/

#proxmox #selfhosted #homelab

8:45pm ET – let's reboot the physical host, I have access to it and a monitor and keyboard so nothing too dicey.Issue a `reboot now`...

8:45pm ET – let's reboot the physical host, I have access to it and a monitor and keyboard so nothing too dicey.

Issue a `reboot now` command via SSH. Things start churning.

8:50pm ET – I'm watching the reboot progress via a monitor physically attached to the box. It's...not...smooth.

Every so often the logs will mention “ZFS pool uncorrectable I/O error”. Damn, not good.

2/

#proxmox #selfhosted #homelab

This was a new one.8:30pm ET – some self-hosted services are running oddly.8:31pm ET – ssh in, wow 400 load, that's a lot, let's...

This was a new one.

8:30pm ET – some self-hosted services are running oddly.
8:31pm ET – ssh in, wow 400 load, that's a lot, let's dig into which VM's are going nuts.
8:35pm ET – start attempting to shut down some VMs. Some shutdown gracefully, some require a stop or reset (kvm/proxmox).

At this point I'm thinking ok, may need to reboot the entire physical box since I'm making no headway on bringing down load by trying and sometimes shutting down VMs.

1/

#proxmox #selfhosted #homelab

I wrote out my notes about setting up Jellyfin in an *unprivileged* container on...

I wrote out my notes about setting up Jellyfin in an *unprivileged* container on Proxmox.

https://mattedwards.org/2023/11/jellyfin-in-an-unprivileged-lxc-with-intel-quicksync/

#blog #jellyfin #proxmox #lxc #homelab #selfhosting

There are two ends to the spectrum of what education is. On the one extreme, education teaches a specific skill - think woodworking,...

There are two ends to the spectrum of what education is.

On the one extreme, education teaches a specific skill - think woodworking, plumbing, electrical work.

On the other extreme, education prepares us for what we don't know and what the future might hold.

Every class you take in college, every book you read, every thing you pack into your brain falls somewhere on that spectrum.

And it’s all valuable.

Quoting @inthehands: https://hachyderm.io/@inthehands/110390815748455836 #retoot

⚠️ PSA about Bookmarks 📖Didn’t realize that if I Bookmark a post from another #Mastodon server, it’ll be removed from my server...

⚠️ PSA about Bookmarks 📖

Didn’t realize that if I Bookmark a post from another #Mastodon server, it’ll be removed from my server after the retention period I’ve defined for my server.

So if I purge everything cached by my server after 7 days, that *also* purges any post I bookmarked more than 7 days ago. (Second screenshot shows that only something I bookmarked 6 days ago is left, and I’ve certainly bookmarked other things!)

More here: https://strangeobject.space/@esther/109937056590192025