Rack Sever Cluster,

Nice :slight_smile:

Motherboard and cpu have now arrived.

Xeon CPU not Core i7.

Hard drive enclosures are now attached to case.


Now I need to figure out a low profile mounting solution for the motherboard

https://www.jaycar.com.au/6-35mm-adhesive-pcb-standoffs-pk-25/p/HP0760 is what I tried but their seems to be a lot of daylight under the motherboard

My board is now booting.

I have no idea though how to put uboot on their or get network booting working.

Okay I don’t have the boards you’re using, so this is all theoretical… but most single-board computers have a port of U-Boot available. Some use RedBoot, which is older and a pain in the arse to get working (needs an OABI toolchain where as the kernel requires EABI), but this has fallen out of favour.

U-Boot will need to be configured and compiled specifically for the board you are using. Even one for a different board using the same SoC is not guaranteed to work since there’s usually lots of ways peripherals and RAM can be hooked up.

For boards that are capable of booting from SD cards, there’ll be instructions on how to place the U-Boot binary on the SD card. For Freescale i.MX286 for example required you to use a special partition type as the first partition, to which you DD-ed a specially prepared image. (https://bne.vrt.com.au/technologicsys/ has some notes on the topic. Fun times, figuring out how to initialise DDR2 memory.)

Raspberry Pi actually is boot-loaded by the GPU, and oddly enough, the GPU is able to read files from FAT32 partitions, you drop a compiled U-boot binary on the first partition with the correct name. In Slack you mentioned you were using the Banana Pi which is the Sun Xi platform…

Those are links that are worth looking at. Debian ships a toolchain for building ARMHF Linux kernels, and it’ll be this same toolchain that you use to build the boot-loader. Gentoo’s crossdev is also pretty good at building such kernels… or a third option, is to grab a tarball of the GNU ARM toolchain from https://developer.arm.com/open-source/gnu-toolchain/gnu-rm.

U-boot itself is configured using a terminal interface and supports rudimentary scripting. So once booted, you’ll be using setenv commands to “configure” environment variables, one of which the boot-loader passes to the run command to execute as a script. U-Boot usually ships with one for network booting.

In essence, the boot procedure will require a DHCP and TFTP server… the procedure will likely work as follows:

  1. On-chip bootloader loads U-Boot first-stage bootloader off the SD card into SRAM
  2. U-Boot first stage initialises DRAM, then loads the rest of U-Boot into DRAM and executes it
  3. U-Boot will display a prompt before executing a default script (which will be your network boot instructions)
  4. Network boot script first brings up the Ethernet interface, acquires an IP address via DHCP, figures out where the TFTP server is and what file to read from the DHCP server reply (aka next-server and filename in ISC DHCPd). The file is your operating system kernel.
  5. Network boot script requests the boot file (kernel) from the TFTP server, downloading it to RAM
  6. U-Boot then boots the downloaded file, passing in any kernel command line options defined in the U-Boot environment
  7. Kernel starts booting, from here, two possibilities:
    7.1 Traditional set-up using Root-over-NFS: Kernel brings the network interface up via DHCP, makes contact with a NFS server then mounts that as its / filesystem. The kernel then executes /sbin/init unless overridden with init=/path/to/alternate/init on the kernel command line.
    7.2 Modern approach using initramfs: Kernel image embeds a root filesystem which is held in RAM and contains the necessary tools to boot the system. Kernel will execute /init.

I’ve done both, option 7.2 typically for things like the Gentoo/MIPS netboot images, option 7.1 for my own stuff. 7.1 has the advantage that the image occupies little RAM on the target systems and allows you to make changes to the files on the host and see those changes on the target, it’s also quick to set up.

7.2 has the advantage that the target is then completely stand-alone and can continue operating without the host.

Kernel boot server

Regardless of where the root lives, you still need to get the kernel onto the device. My recommendation is to look at ISC’s DHCP daemon and tftp-hpa. The Gentoo/MIPS handbook covers setting this up… the concepts given here applies equally to devices running U-Boot as it does to ancient SGI kit:

https://wiki.gentoo.org/wiki/Handbook:MIPS/Installation/Media#Netbooting_overview

If you look at the examples for the SGI hardware, you’ll see next-server and filename… U-Boot will be loading the file specified in filename from the server given in next-server via TFTP.

Setting up Root over NFS

The full gory details can be found in the Linux kernel documentation.

As far as I know, even though the kernel supports NFSv4, Root-over-NFS needs NFS v3. You’ll want a root filesystem stashed somewhere on your boot server, traditionally this was /nfsroot/${HOSTNAME}, but it can be theoretically anywhere. You’ll need the NFS server tools (net-fs/nfs-utils on Gentoo, nfs-kernel-server on Debian) installed.

In /etc/exports, add a line:

/path/to/nfs/root *(ro,no_root_squash,sync,no_subtree_check)

Within /path/to/nfs/root, you’ll need your root filesystem. This could be an unpacked Gentoo Stage 3 tarball (I’ve done that countless times to bootstrap i386, AMD64, MIPS and ARM systems), a debootstrap-generated root filesystem, or even the contents of the root partition your single board computer’s SD card image.

Since all nodes will be mounting the same root FS, you probably don’t want to have them all execute /sbin/init, but rather, will want to give init=/path/to/some/script. This script will look something like the following:

#!/bin/sh -e

# Create a RAMFS volume for storing local system state.
/bin/mount none /mnt/ramfs -t ramfs

# Create a copy of `/etc` and `/var` from our NFS root.
/bin/tar -C / -cf - etc var | tar -C /mnt/ramfs -xf -

# Bind-mount our RAM-backed /etc and /var
/bin/mount /mnt/ramfs/etc /etc -o bind
/bin/mount /mnt/ramfs/var /var -o bind

# Now start /sbin/init as normal
exec /sbin/init

Your kernel command line would then look something like this: root=/dev/nfs ip=dhcp nfsroot=${SERVER_IP}:/path/to/nfs/root init=/usr/local/sbin/init.sh

Other options include using AUFS… mounting partitions from an SD card… all sorts of options.

My recommendation here is just start with one node and drop the init=… stuff first off, play around a bit, then try authoring a script to use for your init script and get it right, then finally bring in more nodes.

Embedded initial filesystem

Once again, the Linux Kernel documentation has some detail on what this is and how it works. I think U-Boot does support loading a separate initrd image, but bundling it all in one file does save you from having to handle multiple files.

The tricky bit is getting the kernel driver modules into the image, and keeping sizes down. You really want the bare minimum, a full Raspbian NOOBS image just isn’t going to fly, I’d be considering building one using buildroot or using something tiny like the Alpine Linux mini root.

You’ll want to unpack that somewhere and specify that in the kernel configuration. Then to build the kernel and image, you run:

$ make zImage modules
$ make modules_install firmware_install INSTALL_MOD_PATH=/full/path/to/your/root/directory
$ make zImage

The first zImage builds all the requisite parts needed to build the modules and kernel. Second command installs the modules and firmware blobs into the to-be-packaged root FS image. Third will generate a zImage that embeds that image.

From here, the image will need the smarts to be able to fetch everything else it needs. As I say, when using initramfs, the kernel instinctively looks for /init, which is a shell script like the above example for NFS root. Gentoo provides one such example here:

https://wiki.gentoo.org/wiki/Custom_Initramfs#Init

Anyway, that should give you some pointers… if you get stuck, give one of us a holler and I’m sure we can assist further.

Thanks.

I will probably have to set up the computer to netboot from first.

Almost had it, the heatsink was too big to fit in the case. I might have to bring it in on Tuesday.

Not sure if you bought your RAM already but might want to have a look here:
http://www.servermonkey.com

Nice site… although one gripe I have is they don’t tell you if they’re DIMMs or SO-DIMMs… ECC SO-DIMMs are rare beasts, but smaller servers require them. I’ll keep them in mind if I ever need the full-size DIMMs though.

Could not find ddr4 ecc on the site.

Wish i knew before buying the cpu though.

Question. I have used 5.25 enclosures for the hard drives. Is their any resources on building your own hard drive mounts?

I am looking for a bit more physical space then the enclosures currently allow

Update: I now have the shift register ICs (74hc595) for controlling the board power and reset switches.

I accidentally ordered surface mount though.

How easy is it to make a board for them?

@catprog.- Quite easy to make a PCB. You just send an email to Elecrow/Seed/DityPCB/BatchPCB.

If you want to make one at home it is a bit trickier and costs a bit more, but also vary easy. Maybe 1/2 hours work for a simple single sided one.

What surface mount package?

I am not sure. The seller had labelled it wrong.

Hmmm… eBay purchase? Surely you’d be able to send it back for a refund as it wasn’t what was advertised.

Perhaps the list here might help identify which one it is? You can get little adaptor boards for most packages and some can be dead-bugged.

I could, but for $1.76(for 20) I don’t think it is worth it.