August 2021. Sam Rushing.
If you're like me, you've tried several times to get a working system built on the rpi4. Eventually you gave up in frustration. In my case, I could not get it to boot off of a usb3 hard drive. It's a weird system, and really not appropriate for any kind of server.
So you're looking for an affordable arm64 system with a little kick.
Solid-Run sells the MacchiatoBIN line, which consists of the 'Single Shot' and the 'Double Shot', in a mini-itx form factor. The Single runs at 1.6GHz, and is missing copper PHYs for the SFP+ ports. The Double runs at 2.0GHz, and has them. If I were to buy again, I would probably go for the Double.
It's a quad-core arm64 system built around the Marvell 8040 SoC.
The Single was $349. Shipping to California was $61 (!) since it's sent from Israel. I think it took about two weeks to arrive.
The board came with a mini-usb for access to the serial console. No documentation.
I'm on macos, so this is a good way to access it:
$ screen /dev/tty.usbserial-DM02XSRS 115200
You can also use minicom.
Mine came with an empty eMMC. The SPI flash had u-boot loaded. After some struggles, I was able to get linux installed to the eMMC, just to verify everything was working.
The wiki has been down for several weeks, and the 'contact' web ui on the website fails to send a message. The 'forum' software is filled with spam, to the point where it's hard to find useful info.
HOWEVER, the wayback machine provides clumsy access to the wiki for now: you can find the page you're looking for via google, and then feed that URL to the wayback machine.
There are quite a few choices for booting this system. There are jumpers on the board for boot choice: SPI, eMMC, SD. At one point I thought I'd bricked it when I messed up loading the SPI flash, but after changing the jumpers to SD card I was back up and running.
There's an alternative oops-bricked-it recovery mode where you can boot over the UART, but I was unable to get this to work. It uses a wake-up pattern followed by a (modified?) xmodem protocol to send an image to boot. Turns out this wasn't needed, since you can boot from a fresh SD card and fix everything from there.
The board supports either u-boot or UEFI. I don't think I was able to boot FreeBSD with u-boot, but the UEFI option works great.
For a case I used the Cooler Master Elite 130 which just has room for 3 3.5" SATA drives. (at 3 different angles!)
Power supply is EVGA 450 W3.
The board came with a 4G DDR4 Non-ECC RAM (samsung m378a5244cb0-ctd), but I believe it's supposed to support ECC. I replaced that with an 8GB G.SKILL stick I had laying around. No issues with either.
I've had the best luck with installing FreeBSD 13.0-RELEASE from the 'disc1' image.
Steps:
dd) to write disc1.iso to a usb fob.After installation, you should see a 'FreeBSD' entry in the UEFI boot menu, and
the machine should boot automatically without user intervention. [note: you can examine and update these boot entries using efibootmgr]
Once I had that working I flashed the UEFI image to the SPI Flash and changed the jumpers, now I don't need to rely on an SD card for booting.
There's 8GB of eMMC on board. I currently have a copy of linux on there, but I think I might install freebsd to have a nice rescue system in place without needing a fob.
The board has two SFP+ 10Gbe ports and a 1Gbe RJ45, none of these seem to be supported by
FreeBSD-13.0-RELEASE. However, I see activity related to the board going on in -CURRENT,
it's possible that support is being worked on. [note: A snapshot build from late August 2021
seems to recognize the devices but did not provide an interface.]
I was able to network the machine using a $15 DGE-560T from D-link in the PCIe slot.
Note: the information on freebsd.org about this card is incorrect, the chipset on board
needs the re driver, not the msk driver.
I was also able to get a USB wifi fob working via the run driver.
make buildworld -j 6 takes about 5 hours. make buildkernel -j 6 about 1400 seconds.
This is with 8GB of RAM.
I installed the system using ZFS raidz1, using two 4TB HGST drives and a 10-year old WD drive.
My plan was to order a third 4TB HGST and practice handling a drive failure. However, I think
I made a mistake in that the old 640G drive was disk 0, and this was the only drive with the
EFI boot system on it. I believe that using gpart bootcode it should be possible to make those
other EFI partitions bootable (after putting a FAT filesystem and the efi/boot/bootaa64.efi file
in place)
The zroot install setup makes identical partitions on all 3 drives, and I (wrongly?) assumed that since each drive had an EFI partition each drive was populated and bootable. As best I can tell those EFI partitions are all zeroes.
At first I did a zpool offline command to deliberately take the 640G out of the pool. This
leaves zfs in a 'degraded' state, but it operates fine, and survives a reboot in that state.
When the 3rd 4TB drive came, I removed the 640G and put the new drive in its place. Once all back together, freebsd was now unbootable.
Recovery:
disc1.iso usb fobgpart backup ada1 | gpart restore -F ada0zpool import to load the degraded pool.zpool replace ada0p3 ada0p3# zpool status
pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Aug 25 16:25:22 2021
579G scanned at 1.06G/s, 126G issued at 237M/s, 579G total
41.8G resilvered, 21.75% done, 00:32:40 to go
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
replacing-0 DEGRADED 0 0 0
ada0p3/old OFFLINE 0 0 0
ada0p3 ONLINE 0 0 0 (resilvering)
ada1p3 ONLINE 0 0 0
ada2p3 ONLINE 0 0 0
errors: No known data errors
#
When the resilvering finished, everything seemed fine but the pool was
massively undersized for the disks attached. I needed to issue a command
to tell it to 'expand': zpool online -e zroot ada0p3
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 391G 6.66T 128K /zroot
zroot/ROOT 7.04G 6.66T 128K none
zroot/ROOT/default 7.04G 6.66T 7.04G /
zroot/tmp 176K 6.66T 176K /tmp
zroot/usr 383G 6.66T 128K /usr
zroot/usr/home 381G 6.66T 381G /usr/home
zroot/usr/ports 1.85G 6.66T 1.85G /usr/ports
zroot/usr/src 854M 6.66T 854M /usr/src
zroot/var 2.64M 6.66T 128K /var
zroot/var/audit 128K 6.66T 128K /var/audit
zroot/var/crash 128K 6.66T 128K /var/crash
zroot/var/log 1.95M 6.66T 1.95M /var/log
zroot/var/mail 197K 6.66T 197K /var/mail
zroot/var/tmp 128K 6.66T 128K /var/tmp
I followed this guide to install netatalk/afp and samba4. I can now select the NAS as a Time Machine backup target, though I haven't yet tested it.
I installed minidlna and have successfully used it to browse a
several-hundred-GB photo collection from my TV.