Creating FreeBSD images in Google Compute

I found various instructions to create a FreeBSD image in Google Compute but non of those instructions where complete of functional.  Here are the correct instructions.

Read more »

Configure SSL Termination with Varnish Caching and HTTP/2

To create the fastest web pages possible in a LAMP stack can be a bit tricky.  This document covers configureing blazing fast HTTPS sites on Freebsd with Apache, Varnish, Nginx, PHP, Mysql and letsencrypt.sh.  Of course you can swap out the PHP/Mysql parts as needed, or even use your own certificates instead of lets encrypt.sh as you need.

Read more »

Mass rename files in Midnight Commander

Here is another script for Midnight Commander to allow for mass renaming of flagged files. To use it you will need to add it to your MC menu file in /etc/mc/mc.menu or ~/.mc/menu.

+ t t
e       Rename tagged files
        set %t; CMD=%{Enter regex}
        while [ -n "$1" ]; do
          echo rename -\"s"$CMD"e\" "$1"
          rename -\"s"$CMD"e\" "$1"
          shift
        done

first you will need to flag the files toy want to rename then execute the user operation. When you do so you will be prompted for the regular expression you with to execute.

Some examples:

 /oldword/newword/   # Change a word.
 /_/ /g   # Replace all _'s with spaces.
 /_\[.*\]//   # Remove _[CRCHASH] tags.
 /\(.*\)//   # Remove (...) sections.

Comapring files with md5 Checksums in Midnight Commander

I wrote a few scripts for Midnight Commander to easily compare multiple copies of a directory via checksums.  There are a few variations listed below along with a bash script that does the same thing.

---

First off is this simple bash script that can be used stand alone but is also required for the MC menu entries. 

/usr/local/bin/md5compare

#!/usr/local/bin/bash
echo -n "Comparing $1 to $2: "
if [ ! -e "$1" ] || [ ! -e "$2" ]
then
 echo File Missing
 exit 1; 
fi

if [ $(md5 -q "$1") = $(md5 -q "$2") ]
then
 echo Files Match
 exit 0;
else
 echo Files Different
 exit 1;
fi

You may need to tweak the bash and script paths for your installation. Also remember to make it executable.

sudo chmod +x /usr/local/bin/md5compare

To execute this command directly just run the following.

md5compare file1 file2

---

Next up are two comparison scripts that can be added MC's user menu.  To do this append them to the mc.menu file in your system /etc or users ~/.mc/menu.

This script will compare the currently highlighted file with one of the same name in the other panel.

+ t n
w       Compare files via MD5 checksum
        /usr/local/bin/md5compare %f %D/%f
        echo Press any key to continue
        read key

This script on the other hand will compare all tagged files to any matching files in the other panel.

+ t t
W       Compare tagged files via MD5 checksum
        for i in %t
        do 
          /usr/local/bin/md5compare "$i" %D/"$i"
        done
        echo Press any key to continue
        read key

Finally here is a menu item to simply dispaly the checksum of the selected file.

+ t n
s       Calculate MD5 checksum
        echo Calculating checksum of %f
        md5 -q %f
        echo Press any key to continue
        read key

Backup your FreeBSD system configuration.

I set up a simple script to create a configuration backups of my FreeBSD box and I thought I would share it. Note that this script will only back up the /etc and /usr/local/etc directories and weighs in at just under 1MB per backup.

First create a backup script as we can't execute our complex command directly in cron.  You may want to customize the exclude options to your licking, the two listed exclusions are the rather large gconf defaults, witch is not needed, and the working files for transmission.

sudo vim /usr/local/sbin/backup-config
bash -c 'tar -czf /root/freebsd-cfg-`date "+%Y-%m-%d"`.tgz --exclude={etc/gconf,usr/local/etc/transmission/home/{resume,torrents,Downloads,blocklists}} /etc/ /usr/local/etc/'

Now make it executable.

chmod +x /usr/local/sbin/backup-config

Now add the job to cron and set it to run weekly as root.

sudo vim /etc/crontab# Backup the entire server configuration once a week.
0 1 * * 0 root backup-config 2>/dev/null

Mixing binary backage and source ports install sources with portupgrade in FreeBSD

Ok so a few weeks ago I made this post about automating update checks and downloads.  While as it turns out there was another issue I ran into.  The problem is simply this:

When using portupgrade -Pa to to FreeBSD updates from packages all updated packages will be installed even if you compile one of the updated packages was compiled locally from the ports tree with custom options or libraries.  This causes any customizations to be lost and possibly version mismatches with system libraries.

The only ways around this problem was to either only upgrade some subset of the ports or to set HOLD_PKGS and manually update problem ports after a package update.  Neither of these solutions is optimal so I took the time to make a patch for portupgrade.

The patch adds a USE_PORTS configuration option that specifies a lit of packages to always build from the ports tree even when portupgrade -P is called.  (The -PP option still works the same as before.)  Thus any custom packages can be added to this list and then an automatic update can be done that will favor binary version when available and when not overridden.  This also means that if one package is built from source its parents and children will still come from binary packages if available.

Build the pached version
Patch, build, and install.

cd /usr/ports/ports-mgmt/portupgrade
sudo make extract
cd work
sudo wget http://pynej.dnsalias.com/Shared/pkgtools-2.4.6-1.patch
sudo patch -p0 < pkgtools-2.4.6-1.patch
cd pkgtools-2.4.6
sudo make isntall clean

You may also want to add portupgrade to your HOLD_PKGS list in /usr/local/etc/pkgtools.conf so it doesn't get replaced in the future.

Configuration
To configure the list of overrides you will need to edit the following section:

sudo vim /usr/local/etc/pkgtools.conf
USE_PORTS = [
    
  ]

Here is a sample of some common overrides.

•  php5 to enable the apache module
•  php5-mysql to get the binding to the correct version of mysql-server
•  mod_perl2 to enable apache module
•  phpmyadmin as it depends ont he wrong php5-mysql paskage)

sudo vim /usr/local/etc/pkgtools.conf
USE_PORTS = [
    'lang/php5',
    'databases/php5-mysql',
    'databases/phpmyadmin',
    'www/mod_perl2',
  ]

Configure VirtualBox Deamon

A while back I switched from VMware to VirtualBox(when switching to freebsd) and ran into a problem getting virtual machines to start on server boot. Virtual box supports background operation(headless mode) but freebsd does not come with and rc service for doing so. Included below is a freebsd version to accomplish startup/shutdown task as well as other common operations.

To install save this script into /usr/local/etc/rc.d/virtualbox and make it executable. Then add the following lines to your /etc/rc.conf as needed.

# Load the VirtualBox drivers and start any default machines.
vboxnet_enable="YES" # Enable drivers from package virtualbox-ose-kmod.
virtualbox_enable="YES"
virtualbox_autostart="ASG" # List of machine names to start on boot.
virtualbox_user="vboxuser"
virtualbox_group="vboxusers"

A few other notes: I actually set up a separate account for VirtualBox and the daemon supports this. Also please note that the autostart list is done by name and the names can not have spaces. You can also manually start and stop machines with the startvm and stopvm options.

/usr/local/etc/rc.d/virtualbox
#!/bin/sh
#

# PROVIDE: virtualbox 
# REQUIRE: LOGIN vboxnet

#
# Add the following lines to /etc/rc.conf to enable the virtualbox daemon:
#
# virtualbox_enable="YES"
# virtualbox_autostart="BOX1 BOX2"
#


. /etc/rc.subr

name="virtualbox"
rcvar=`set_rcvar`
extra_commands="status startvm stopvm"

start_cmd="${name}_start"
stop_cmd="${name}_start"
status_cmd="${name}_status"
startvm_cmd="${name}_startvm"
stopvm_cmd="${name}_stopvm"
poweroffvm_cmd="${name}_poweroffvm"

load_rc_config $name

virtualbox_user=${virtualbox_user:-"vboxuser"}
virtualbox_group=${virtualbox_group:-"vboxusers"}
virtualbox_autostart=${virtualbox_autostart:-""}

SU="su $virtualbox_user -c"
VBOXMANAGE="/usr/local/bin/VBoxManage -nologo"

virtualbox_start()
{
        echo "Starting ${name}."

        echo $virtualbox_autostart | awk -F , '{for(i=1;i<=NF;i++) print $i}' |  while read VM; do
            $SU "$VBOXMANAGE startvm \"$VM\" --type headless" 
        done
}

virtualbox_stop()
{
        echo "Stopping ${name}."
        $SU "$VBOXMANAGE list runningvms" | sed 's/.*"\(.*\)".*/\1/' | while read VM; do
            $SU "$VBOXMANAGE controlvm \"$VM\" acpipowerbutton"
        done

        wait_for_closing_machines
}

virtualbox_startvm()
{
        $SU "$VBOXMANAGE startvm \"$*\" --type headless"
}

virtualbox_stopvm()
{
        $SU "$VBOXMANAGE controlvm \"$*\" acpipowerbutton"
}

virtualbox_poweroffvm()
{
        $SU "$VBOXMANAGE controlvm \"$*\" poweroff"
}

virtualbox_status()
{

        $SU "$VBOXMANAGE list runningvms" | sed 's/.*"\(.*\)".*/\1/' | while read VM; do
            echo "$VM "
        done
}

wait_for_closing_machines() {
    RUNNING_MACHINES=`$SU "$VBOXMANAGE list runningvms" | wc -l`
    if [ $RUNNING_MACHINES != 0 ]; then
        sleep 5
        wait_for_closing_machines
    fi
}

run_rc_command "$@"

Setting up FreeBSD to Auto-Download and Notify Updates

03/11/2010- Update: Updated the portsnap command to properly apply the port updates. As configured before the changes would show up but the cron task to download packages would never download them.

Overview
The default install of FreeBSD is very stable, and works well. And it is relatively simple manually do updates, but I recent looked into setting up auto-updates like Ubuntu does. The way I have it et up now is the following.

Automatic downloading of kernel/world updates to the FreeBSD release.
Automatic downloading and updating of the current ports tree.
Automatic downloading of any updated binary packages of installed ports.

These three tasks are set as cron jobs and run once a day/week to check for and download updates. Reports are also sent to the root account on checking so you will be notified where updates are available. The administrator can then manuals install the system updates, binary packages updates, and source port updates from the local cache.

Setup
First off we will add a few cron jobs to auto-download our updates. Add these lines to /etc/cron and customize the run time as desired.

# Check for freebsd updates, download them, and mail root.
0       2       *       *       0       root    freebsd-update cron
# Check for ports updates, download them, and mail root.
0       3       *       *       0       root    portsnap cron update && pkg_version -vIL=
# Check for binary pacakge updates, download them, and mail root.
0       4       *       *       0       root    portupgrade -PFa

To enable the email reports you need to add an alias to send mail to forward root's mail to an administrator. To do so edit the file /etc/aliases and add line like so with your username.

root: adminaccount

Then run the following command to make the change take effect.

cd /etc/mail && sudo make aliases

You will also need to install the portutils package if you don't have it for package updating.

cd /usr/ports/ports-mgmt/portupgrade && sudo make install

Once installed we need to change the package source location to pull binary package updates from the stable branch instead of the release branch.  The release packages are never updated and as such we would never find binary updates. To change this edit the /usr/local/etc/pkgtools.conf file and change the PKG_SITES variable to the following.

PKG_SITES = [
    sprintf('ftp://ftp.freebsd.org/pub/FreeBSD/ports/%s/packages-%s-stable/', OS_PLATFORM, OS_MAJOR)  
  ]

Unfortunately the  portupgrade utility does not respect packages you customized and build by hand and will just overwrite them with the binary version.  To get around this you can add any exceptions you want to HOLD_PKGS array in this file and update them manually.  You way also want to add any languages you don't use to the IGNORE_CATEGORIES array at this time as well to speed up the ports commands.

Manual Update
Once all these steps are done we can force a manual update of all three with the following commands, though they will take a bit to complete.

sudo freebsd-update fetch
sudo portsnap fetch update
sudo portupgrade -PFa

Installing Updates
If using ZFS you may want to make a snapshot first.

sudo zfs snapshot zroot@ver-date
sudo zfs snapshot zroot/usr@ver-date

When you want to do an actual update to the system here are the commands to install the downloaded updates.

sudo freebsd-update install
sudo portupgrade -Pa

And finally its a good idea to clean out the old files manually or via another cron task..

portsclean -CDPL

If everything went smoothly you may wish to remove the old snapshots.

sudo zfs destroy zroot@ver-date
sudo zfs destroy zroot/usr@ver-date

Set up the transmission-deamon BitTorrent Client in FreeBSD

I switched over to FreeBSD a while back and still am in the process of configuring it the way my Ubuntu server used to be set up. One of the things I used to have was uTorrent running, via wine, and auto-loading torrents from a shared directory. This allowed me to queue downloads from anywhere and have them auto-download. It is not possible to set up my 32bit FreeBSD box the same was as ZFS prevents wine from operating. (This is a known issue due to the custom kernel with an increased memory map.)

As such I was looking for a similar solution that would run as a daemon and could automatically process torrent files. And as it turns out the transmission-daemon package does exactly that. Following is a brief review of setting up this service and some customizations I did.

Read more »