Making a Linux home server sleep on idle and wake on demand — the simple way

2023-04-16 — linux, backup, networking, wireshark, ruby, wake-on-lan, efficiency, homelab — 18 min read

It began with what seemed like a final mundane touch to my home server setup for hosting Time Machine backups: I wanted it to automatically sleep when idle and wake up again when needed. You know, sleep on idle — hasn't Windows had that built in since like Windows 98? How hard could it be to configure on a modern Ubuntu install?

To be fair, I wanted more than just sleep on idle, I also wanted wake on request — and that second bit turns out to be the hard part. There were a bunch of dead ends, but I stuck out it to find something that "just works" without the need to manually turn on the server for every backup. Join me on the full adventure further down, or cut to the chase with the setup instructions below.

Heads up: There's a newer, improved version of this guide that's also open for community contributions. Read the latest version »

tl;dr

Outcome:

Server automatically suspends to RAM when idle
Server automatically wakes when needed by anything else on the network, including SSH, Time Machine backups, etc.

You'll need:

An always-on Linux device on the same network as your server, e.g. a Raspberry Pi
A network interface device for your server that supports wake-on-LAN with unicast packets

On the server:

Enable wake-on-LAN with unicast packets (not just magic packets), make it persistent

sudo ethtool -s eno1 wol ug
sudo tee /etc/networkd-dispatcher/configuring.d/wol << EOF
#!/usr/bin/env bash

ethtool -s eno1 wol ug || true
EOF
sudo chmod 755 /etc/networkd-dispatcher/configuring.d/wol

Set up a cron job to sleep on idle (replace /home/ubuntu with your desired script location)

tee /home/ubuntu/auto-sleep.sh << EOF
#!/bin/bash
logged_in_count=$(who | wc -l)
# We expect 2 lines of output from `lsof -i:548` at idle: one for output headers, another for the 
# server listening for connections. More than 2 lines indicates inbound connection(s).
afp_connection_count=$(lsof -i:548 | wc -l)
if [[ $logged_in_count < 1 && $afp_connection_count < 3 ]]; then
  systemctl suspend
else
  echo "Not suspending, logged in users: $logged_in_count, connection count: $afp_connection_count"
fi
EOF
chmod +x /home/ubuntu/auto-sleep.sh
sudo crontab -e
# In the editor, add the following line:
*/10 * * * * /home/ubuntu/auto-sleep.sh | logger -t autosuspend

Disable IPv6: this approach relies on ARP, which IPv6 doesn't use

sudo nano /etc/default/grub
# Find GRUB_CMDLINE_LINUX=""
# Change to GRUB_CMDLINE_LINUX="ipv6.disable=1"
sudo update-grub
sudo reboot

Optional: Configure network services (e.g. Netatalk) to stop before sleep to prevent unwanted wakeups due to network activity

sudo tee /etc/systemd/system/netatalk-sleep.service << EOF
[Unit]
Description=Netatalk sleep hook
Before=sleep.target
StopWhenUnneeded=yes

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/bin/systemctl stop netatalk
ExecStop=-/usr/bin/systemctl start netatalk

[Install]
WantedBy=sleep.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable netatalk-sleep.service

On the always-on device:

Install ARP Stand-in: a super simple Ruby script that runs as a system service and responds to ARP requests on behalf of another machine. Configure it to respond on behalf of the sleeping server.
Optional: Configure Avahi to advertise network services on behalf of the server when it's sleeping.

sudo apt install avahi-daemon
sudo tee /etc/systemd/system/avahi-publish.service << EOF
[Unit]
Description=Publish custom Avahi records
After=network.target avahi-daemon.service
Requires=avahi-daemon.service

[Service]
ExecStart=/usr/bin/avahi-publish -s homeserver _afpovertcp._tcp 548 -H homeserver.local

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable avahi-publish.service --now
systemctl status avahi-publish.service

Caveats

The server's network device needs to support wake-on-LAN from unicast packets
To prevent unwanted wake-ups, you'll need to ensure no device on the network is sending extraneous packets to the server

How I got there

First, a bit about my hardware, as this solution is somewhat hardware-dependent:

HP ProDesk 600 G3 SFF
CPU: Intel Core i5-7500
Network adapter: Intel I219-LM

Sleeping on idle

I started with sleep-on-idle, which boiled down to two questions:

How to determine if the server is idle or busy at any given moment
How to automatically suspend to RAM after being idle for some time

Most of the guides I found for sleep-on-idle, like this one, were for Ubuntu Desktop — sleep-on-idle doesn't seem to be something that's commonly done with Ubuntu Server. I came across a few tools that looked promising, the most notable being circadian. In general, though, there didn't seem to be a standard/best-practice way to do it, so I decided I'd roll it myself the simplest way I could.

Determining idle/busy state

I asked myself what server activity would constitute being busy, and landed on two things:

Logged in SSH sessions
In-progress Time Machine backups

Choosing corresponding metrics was pretty straightforward:

Count of logged in users, using who
Count of connections on the AFP port (548), using lsof (I'm using AFP for Time Machine network shares)

For both metrics, I noted the values first at idle, and then again when the server was busy.

Automatically suspending to RAM

To keep things simple, I opted for a cron job that triggers a bash script — check out the final version shared above. So far it's worked fine; if I ever need to account for more metrics in detecting idle state, I'll consider using a more sophisticated option like circadian.

Waking on request

With sleep-on-idle out of the way, I moved on to figuring out how the server would wake on demand.

Could the machine be configured to automatically wake upon receiving a network request? I knew Wake-on-LAN supported waking a computer up using a specially crafted "magic packet", and it was straightforward to get this working. The question was if a regular, non-"magic packet" could somehow do the same thing.

Wake on PHY?

Some online searching yielded a superuser discussion that looked particularly promising. It pointed to the man page for ethtool, the Linux utility used to configure network hardware. It shared ethtool's complete wake-on-LAN configuration options:

wol p|u|m|b|a|g|s|f|d...
      Sets Wake-on-LAN options.  Not all devices support
      this.  The argument to this option is a string of
      characters specifying which options to enable.

      p   Wake on PHY activity
      u   Wake on unicast messages
      m   Wake on multicast messages
      b   Wake on broadcast messages
      a   Wake on ARP
      g   Wake on MagicPacket™
      s   Enable SecureOn™ password for MagicPacket™
      f   Wake on filter(s)
      d   Disable (wake on nothing).  This option
          clears all previous options.

It pointed in particular to the Wake on PHY activity option, which seemed perfect for this use-case. It seemed to mean that any packet sent to the network interface's MAC address would wake it. I enabled the flag using ethtool, manually put the machine to sleep, then tried logging back in using SSH and sending pings. No dice: the machine remained asleep despite multiple attempts. So much for that 😕

Breakthrough: wake on unicast

None of ethtool's other wake-on-LAN options seemed relevant, but some more searching pointed to the Wake on unicast messages as another option to try. I enabled the flag using ethtool, manually put the machine to sleep, then tried logging back in using SSH. Bingo! This time, the machine woke up. 🙌 With that, I figured I was done.

Not so fast — there were two problems:

Sometimes, the server would wake up without any network activity that I knew of
Some period of time after the server went to sleep, it would become impossible to wake it again using network activity other than a magic packet

A closer look at the same superuser discussion above revealed exactly the reason for the second problem: shortly after going to sleep, the machine was effectively disappearing from the network because it was no longer responding to ARP requests.

ARP

So the cached ARP entry for other machines on the network was expiring, meaning that they had no way to resolve the server's IP address to its MAC address. In other words, an attempt to ping my server at 192.168.1.2 was failing to even send a packet to the server, because the server's MAC address wasn't known. Without a packet being sent, there was no way that server was going to wake up.

Static ARP?

My first reaction: let's manually create ARP cache entries on each network client. This is indeed possible on macOS using:

sudo arp -s [IP address] [MAC address]

But it also didn't meet the goal of having things "just work": I was not interested in creating static ARP cache entries on each machine that would be accessing the server. On to other options.

ARP protocol offload?

Some more searching revealed something interesting: this problem had already been solved long ago in the Windows world.

It was called ARP protocol offload, and it goes like this:

The network hardware is capable of responding to ARP requests independently of the CPU
Before going to sleep, the OS configures the network hardware to respond to ARP requests
While sleeping, the network hardware responds to ARP requests on its own, without waking the rest of the machine to use the CPU

Voila, this was exactly what I needed. I even looked at the datasheet for my network hardware, which lists ARP Offload as a feature on the front page.

The only problem? No Linux support. I searched the far reaches of the internet, then finally dug into the Linux driver source code to find that ARP offload isn't supported by the Linux driver. This was when I briefly pondered trying to patch the driver to add ARP offload... before reminding myself that successfully patching Linux driver code is far beyond what I could hope to achieve in a little free-time project like this one. (Though maybe one day...)

Breakthrough: ARP Stand-in

What I was attempting didn't seem much different from the static IP mapping that's routinely configured on home routers, except that it was for DHCP instead of ARP. Was there no way to make my router do the same thing for ARP?

Some more digging into the ARP protocol revealed that ARP resolution doesn't even require a specific, authoritative host to answer requests — any other network device can respond to ARP requests. In other words, my router didn't need to be the one resolving ARP requests, it could be anything. Now how could I just set up something to respond on behalf of the sleeping server?

Here's what I was trying to do:

I thought it must be possible to implement as a Linux network configuration, but the closest thing I found was Proxy ARP, which accomplished a different goal. So I went one level deeper, to network programming.

Now, how to go about listening for ARP request packets? This is apparently possible to do using a raw socket, but I also knew that tcpdump and Wireshark were capable of using filters to capture only packets of a given type. That led me to look into libpcap, the library that powers both of those tools. I learned that using libpcap had a clear advantage over a raw socket: libpcap implements very efficient filtering directly in the kernel, whereas a raw socket would require manual packet filtering in user space, which is less performant.

Aiming to keep things simple, I decided to try writing the solution in Ruby, which led me to the pcaprub Ruby bindings for libpcap. From there, I just needed to figure out what filter to use with libpcap. Some research and trial/error yielded this filter:

arp and arp[6:2] == 1 and arp[24:4] == [IP address converted to hex]

For example, using a target IP address of 192.168.1.2:

arp and arp[6:2] == 1 and arp[24:4] == 0xc0a80102

Let's break this down, using the ARP packet structure definition for byte offets and lengths:

arp — ARP packets
arp[6:2] == 1 — ARP request packets. [6:2] means "the 2 bytes found at byte offset 6".
arp[24:4] == [IP address converted to hex] — ARP packets with the specified target address. [24:4] means "the 4 bytes found at byte offset 24".

The rest is pretty straightforward and the whole solution comes out to only ~50 lines of Ruby code. In short, arp_standin is a daemon that does the following:

Starts up, taking these configuration options:
- IP and MAC address of the machine it's standing in for (the "target")
- Network interface to operate on
Listens for ARP requests for the target's IP address
On detecting an ARP request for the target's IP address, responds with the target's MAC address

Since the server's IP → MAC address mapping is defined statically through the arp_standin daemon's configuration, it doesn't matter if the Raspberry Pi's ARP cache entry for the server is expired.

Check out the link below to install it or explore the source code further:

arp_standin repository on GitHub

ARP is used in IPv4 and is replaced by Neighbor Discovery Protocol (NDP) in IPv6. I don't have any need for IPv6 right now, so I disabled IPv6 entirely on the server using the steps shown above. It should be possible to add support for Neighbor Discovery to the ARP-Standin service as a future enhancement.

With the new service running on my Raspberry Pi, I used Wireshark to confirm that ARP requests being sent to the server were triggering responses from the ARP Stand-in. It worked 🎉 — things were looking promising.

Getting it all working

The big pieces were in place:

the server went to sleep after becoming idle
the server could wake up from unicast packets
other machines could resolve the server's MAC address using ARP, long after it went to sleep

With the ARP Stand-in running, I turned on the server and ran a backup from my computer. When the backup was finished, the server went to sleep automatically. But there was a problem: the server was waking up immediately after going to sleep.

Unwanted wake-ups

First thing I checked was the Linux system logs, but these didn't prove too helpful, since they didn't show what network packet actually triggered the wakeup. Wireshark/tcpdump were no help here either, because they wouldn't be running when the computer was sleeping. That's when I thought to use port mirroring: capturing packets from an intermediary device between the server and the rest of the network. After a brief, unsuccessful attempt to repurpose an extra router running OpenWRT, a search for the least expensive network switch with port mirroring support yielded the TP-Link TL-SG105E for ~$30.

TL-SG105E: a simple, inexpensive switch with port-mirroring support

With the switch connected and port mirroring enabled, I started capturing with Wireshark and the culprits immediately became clear:

My Mac, which was configured to use the server as a Time Machine backup host using AFP, was sending AFP packets to the server after it had gone to sleep
My Netgear R7000, acting as a wireless access point, was sending frequent, unsolicited NetBIOS NBTSTAT queries to the server

Eliminating AFP packets

I had a hunch about why the Mac was sending these packets:

The Mac mounted the AFP share to perform a Time Machine backup
The Time Machine backup finished, but the share remained mounted
The Mac was checking on the status of the share periodically, as would be done normally for a mounted network share

I also had a corresponding hunch that the solution would be to make sure the share got unmounted before the server went to sleep, so that the Mac would no longer ping the server for its status afterwards. I figured that shutting down the AFP service would trigger unmounting of shares on all its clients, achieving the goal. Now I just needed to ensure the service would shut down when the server was going to sleep, then start again when it woke back up.

Fortunately, systemd supports exactly that, and relatively easily — I defined a dedicated systemd service to hook into sleep/wake events (check out the configuration shared above). A Wireshark capture confirmed that it did the trick.

Eliminating NetBIOS packets

This one proved to be harder, because the packets were unsolicited — they seemed random and unrelated to any activity being done by the server. I thought they might be related to Samba services running on the server, but the packets persisted even after I completely removed Samba from the server.

Why was my network router sending NetBIOS requests, anyway? Turns out that Netgear routers have a feature called ReadySHARE for sharing USB devices over the network using the SMB protocol. Presumably, the router firmware uses Samba behind the scenes, which uses NetBIOS queries to build and maintain its own representation of NetBIOS hosts on the network. Easy — turn off ReadySHARE, right? Nope, there's no way to do that in Netgear's stock firmware 😒.

That led me to take the plunge and flash the router with open-source FreshTomato firmware. I'm glad I did, because the firmware is much better than the stock one anyway, and it immediately stopped the unwanted NetBIOS packets.

Time Machine not triggering wake-up

I was getting close now: the server remained asleep, and I could reliably wake it up by logging in with SSH, even long after it went to sleep.

This was great, but one thing wasn't working: when starting a backup on my Mac, Time Machine would show a loading state indefinitely with Connecting to backup disk... and eventually give up. Was the server failing to wake up from packets the Mac was sending, or was the Mac not sending packets at all?

Time Machine dialog stuck in connecting state

A port-mirrored Wireshark capture answered that question: the Mac wasn't sending any packets to the server, even long after it started to say Connecting to backup disk.... Digging into the macOS Time Machine logs with:

log show --style syslog --predicate 'senderImagePath contains[cd] "TimeMachine"' --info

A few entries made it clear:

(TimeMachine) [com.apple.TimeMachine:Mounting] Attempting to mount 'afp://backup_mbp@homeserver._afpovertcp._tcp.local./tm_mbp'
...
(TimeMachine) [com.apple.TimeMachine:General] Failed to resolve CFNetServiceRef with name = homeserver type = _afpovertcp._tcp. domain = local.

The Mac was using mDNS (a.k.a. Bonjour, Zeroconf) to resolve the backup server's IP address using its hostname. The server was asleep and therefore not responding to the requests, so the Mac was failing to resolve its IP address. This explained why the Mac wasn't sending any packets to the server, leaving it asleep.

mDNS stand-in

I already had an ARP stand-in service, now I needed my Raspberry Pi to also respond to mDNS queries for the server while it slept. I knew that Avahi was one of the main mDNS implementations for Linux. I first tried these instructions using .service files to configure my Raspberry Pi to respond to mDNS queries on behalf of the server. I used the following on the Mac to check the result:

dns-sd -L homeserver _afpovertcp._tcp local

For some reason, that approach just didn't work; Avahi didn't respond on behalf of the server. I experimented instead with avahi-publish (man page), which (to my pleasant surprise) worked right away using the following:

avahi-publish -s homeserver _afpovertcp._tcp 548 -H homeserver.local

With that, I just needed to create a systemd service definition that would automatically run the avahi-publish command on boot (check out the configuration shared above).

🏁 Finish

With all the wrinkles ironed out, everything has been working well now for over a month. I hope you've enjoyed following along and that this approach works for you too.

This post was discussed on Hacker News and Reddit.