Monday, May 19, 2014

Excessive Disk IO/CPU Wait Time on Sophos

I'v used Sophos(formerly Astaro) for years.  Recently though I'v had some performance issues.  I run it in a VM and the system has been slow to respond to new requests as of late.  Logging into the admin interface was even worse.

Rejoice though for I have found the solution.  If the reporting database becomes corrupted from a power failure or something similar the system will still work as expected but the postgresql process will churn away trying to query the invalid data.  This data isn't particularly important as its just the reporting data, and not even all the reporting data at that.  As far as I can tell its mostly used for the Web Protection reporting.

Anyway its a quick fix to reset the database and fix the high Disk IO issue.  Just log into the console and run the following command to remove and recreate the corrupted data.

sudo /etc/init.d/postgresql rebuild

You can run the following command first to verify that this is an issue.  If you see a high cpu-wa value that is constantly over 95 that you have some data corruption issues.  (Its the second to last column.)

vmstat 5

Shared XBMC running from a Raspberry Pi with NFS and MySQL

So I was recently saddened by the death of my first Gen Apple TV.  I had XBMC loaded on there and it would auto-mount my media via NFS shares and connect back to a central database.  I decided to try out the new 35$ Raspberry as an alternate to trying to fix it(it was slow to start up anyway).  I looked at just using a cable to the computer but that was inconvenient.

Result: Perfect Success.  I have had no real issues and it even runs AirPaly now(@TV didn't.)


Heres what I bought/collected.  I had most the cables and things.
  • Raspberry Pi ~$40
    • Note that Model B is required for HD quality video.
  • Tiny USB Wifi ~$10
    • Technically you could just use a network cable but the Wifi works flawlessly at 1080p so no need to.
  • HDMI Cable
    • This cable can honestly be as short as you can find as the Pi can just sit right behind the TV.
  • 5V 1A(min) Micro USB Charger
    • Or USB with a USB to Micro USB cable.
    • I use a Apple USB power block(From iPad Air of iPhone 5) and a cheap o USB to Micro USB cable.  But anything that puts out 5V and at least 1 Amp will work.
  • 4GB SC Card
    • They recomend a Class 10 High Speed card for performance, but honestly the 5 year old card that I left in the car, sat on, and broke the lock button off works just fine and boots right up.  (I have has some file corruption issues but that might just be from unplugging it and not the SD card itself.)
  • Some way to program the card
    • A newer iMac or Pc with a SD card reader, or a some sort of SD Card Reader.
  • Borrow a USB Keyboard
    • You will need a keyboard for all of 2 minutes.  You don't need a mouse for anything.  Once you connect to the Wifi you don't need the keyboard anymore.
  • Remote
    • Don't need one.  Use you Smart Phone or tablet as a free fully functional remote.  (See below)


There are a few pieces of software that we can prepare/setup now.

Operating System

First up the Operating System for the Raspberry Pi.  There are a few choices here but I recommend Xbian as it has all the features we need and worked on my first try.  It is also very speedy.  To set it up follow these directions:

  • Grab the SD card and find a computer you can attach it to.  Make sure to get any files off it before you start because this will destroy all the data on it.
  • Go here and download the installer for the computer/operating system you are programming the SD card on.
  • Plug in the SD card and run the installer(as administrator as needed.)
  • Install the latest version of Xbian onto the SD Card.
  • Make sure to safely eject any volumes when done so nothing get's corrupted.

Remote Software

While that is installing you can get the Remote Control software for your smart phone/tablet to control the Raspberry Pi.
  • MPEG2 Licence ~$5
    • You will need this later to play YouTube, some older types of content, and to receive AirPlay video content. (Though rented movies won't work due to encryption.)
  • iOS - Free
  • Android - Free
  • Other
    • There is a web interface you can use from other devices to control the Raspberry Pi but I don't recommend that.  Instructions for that are included below in the configuration.


Setup is a cinch.  Plug the memory card in. (Make sure it unlocked)  Plug in the Wifi Card and temporary keyboard.  Hook up the HDMI to your TV and change the TV to the proper input.  Then finally plugin in the power and watch the magic happen.


Note: I had a few problems during setup.  When I would reboot form time to time the system would not come back up.  I ended up having to unplug the Pi and stick the memory card back into my desktop and mount/eject it to get it to start up again.  Not that it's working i don't unplug it so I'm not sure if this is still a problem.  I suspect it is due to the poor quality of my SD card.


When booting up your Pi for the first time you will need to configure the network. In the setup wizard go to the network tab and switch to the WLAN interface(wlan0).  Then just switch to DHCP Mode and join your wireless network.  You can set a static IP here but nothing we will use it for will require a static IP and you can access it by the Host Name (xbian.local) as needed.

Note: If the keyboard isn't work try to unplug and plug the keyboard back in.

Now make sure you can connect to the Pi from the network.  On your computer use SSH to connect.
  • On OSX open the Terminal and type in the following command and press enter.
  • ssh xbian@xbian.local
  • On Windows you will need to download Putty and fill in xbian for the usernam, xbian.local for the host name, and select SSH for the type.
In either case you should get prompted for a password which is raspberry.

If you have problems go into Settings > Xbian > Networking and make sure thet you successfully joined the network and have an IP address.

Remote Control

We want to get this working next as it will let us use the phone instead of needing the keyboard.
  • In XBMC go into Settings > Services > Web Server and check Allow control of XBMC via HTTP.
  • Also go into Settings > Services > Zeroconfig and check Announce these services to other systems via Zeroconf.
  • Then on your phone open up the remote control and add a new host.  On an iPhone you can just click Find XBMC and the Pi will show and and configure itself automatically.
  • You may need to manually configure the server on Android devices.
Once the remote is working we won't need the keyboard any more, you can even use the keyboard on your phone to fill in text fields.  See the remote app help for ore details.

Shared XBMC Database

If you wish to use a central database to run XBMC from multiple computers then install MySQL on the file server and create two database's for XBMC.  Then on the Pi you can edit your config to connect to the shared database.  This is not required to use NFS, but rather if you want to use the same meta-data(playes, details, pause point) from multiple systems.

To enter the database information first SSH into the Pi and then run the following.
sudo nano /home/xbian/.xbmc/userdata/advancedsettings.xml
Then inside of the <advancedsettings> add the following filling in the correct values for the host, database, and login details and save the file.


Shared Media with NFS

Now we want to get connected to a central data store.  Personally I run a FreeBSD/ZFS system with Raid5 but any NAS type setup will work.  The performance of Samba/Windows file Sharing is not up to snuff though so we need to use NFS instead.  Look into configuring NFS exports for your operating system and export the media directories.

Now to configure the Pi we can use the XBMC interface but I have had limited success with that option.  Instead I would recommend setting up the mounts in Xbian directly and then adding the media as a local directory in XBMC.

First connect to the Pi in SSH.  Then do the following steps updating/adding mount points as necessary:

Change XBMC to delay starting until the network is fully activated.
sudo nano /etc/init/xbmc-preload.conf
Replace the following line with the next one and save the file.
start on started mount all
start on stopping mountall and started networking and net-device-up wlan0

Manually mount each share to make sure they are working.
For each share you want to connect to from the Pi type in the following and update the directory name, server address, and full server path as applicable.
sudo mkdir /media/video
sudo mount -t nfs /media/video
ls /media/video

Assuming that worked set them to auto-mount at system boot.  Don't worry, the system won't hang if they are not available instead just showing empty directories.
sudo nano /etc/fstab
Add the following line to the end of the file for each share and adjust the directory name, server address, and full server path as applicable and save. /media/video nfs defaults 0 0

Finally we need to make XBMC aware of this new content.  You can do this in XBMC but I find it easier to just update the config file directly.
sudo nano /home/xbian/.xbmc/userdata/sources.xml
Add the following section inside of the <sources> tag and save.  Make sure you don't duplicate the <video> tag if there is already one present but update it to match instead.
  <default pathversion="1"></default>
    <name>Shared Media</name>
    <path pathversion="1">/media/</path>

Note: If you using a shared database and multiple devices then your sources configuration needs to match.  You may need to crete some symlinks to get the media to show up in the same place on all the computers. 


We are to a point now where you can reboot and everything should restart and come up correctly.  You should now be able to play moves and update your library.  If you didn't connect to a existing library you will need to run a full scan which will take some time.

You should be able to restart safely by resting the power but I recommend logging into SSH and run sudo restart the first time just to be safe.

MPEG2 Video

To allow for decoding of some older formats, various internet streams, and AirPlay Video you will need to do the following to get a MPEG-2 License. 

  • SSH into the Pi and run the following command. 
  • cat /proc/cpuinfo | grep Serial
  • Get the serial code from the output and go here to buy a license.
  • In XBMC go to Settings > Xbian > Codecs and enter the code you get from your purchase.  Remember you can use the keyboard on your phone instead of navigating the onscreen keyboard.

Other XBMC Settings

  • Enable AirPlay playback.
    • Settings > Services > AirPlay >
      • Allow XBMC to Receive AirPlay Content: Yes
      • Use Password Protection: No
  • Stack related movies into sets.
    • Settings > Video > Library > Group Movies in Sets: Yes
  • Enable Surround Sound if you have it and DTS and AC3 if your receiver supports them.
    • Setting > System > Audio Output
      • Speaker Configuration
      • DTS Capable Receiver
      • AC3 Capable Receiver

AirPlay Video

To get AirPlay to work constantly you will need to disable IPv6 on xbian.  It is enabled by default so to turn it of logo in via ssh and run the following command.
sudo nano /etc/sysctl.d/disableipv6.conf
Then press add the following line and save the file.

Also you need to make sure that you didn't set a username/password for AirPlay as that won't work.  If so navigate to Settings > Services > AirPlay and uncheck Use Password Protection.


Things that are confirmed working in this setup:
  • H264 up to 1080p
  • MPEG4(divx, xvid, etc) up to 1080p
  • DTS 5.1
  • AC3 5.1
  • Stereo
  • Volume control via Remote App
  • NFS Playback @ 1080p
  • Subtitles (hard and soft)
  • MKV containers up to 1080P
Does not work:
  • There are, sadly, no click sounds when navigating the UI.
  • Some configurations in MKV files that won't even start playing and just come back with a File Unavailable error. Will investigate further
  • FLAC Audio. When do I even have this?


Extra scrapper rules

I have also updated my creepers to support the following folder structure:
  • /Show Name/Season 1/01 Title.mp4
  • /Show Name/Season 1/Show Name 01 Episode Title.mp4
  • /Show Name/Season 1/Show Name 01.mp4
  • /Show Name/Season 1/Show Name S01E01 Episode Title.mp4
  • /Show Name/Season 1/Show Name S02E01.mp4

To do this i have to edit run the following and add the next line.
sudo vi /home/xbian/.xbmc/userdata/advancedsettings.xml
Then inside of the <advancedsettings> add the following section and save the file.
    <regexp>[/\\]Season[\._ ]([0-9]+)[^\\/]*[\\/][^\\/0-9]*[\._ ](?:S?[0-9]+[xE])?([0-9]+)([^\\/]*)</regexp>

Issues with updates

You may need to run the following to get updates working.
apt-get -o Dpkg::Options::="--force-all" upgrade

Offline Backup

You may also want to create a binary backup of the SD Card once configured to quickly restore from if you have any file system corruption issues.  This has bit me more then once so I would recommend it.  Its a quick process.  

Just plug the memory card back into a computer and make a image of the SC card.  On OS X you can do this quickly using Disk Utility.  Select the entire drive(not just the first partition) and click new image to save copy of the entire SD card.  You can later restore the image using the Restore tab.

Tuesday, September 10, 2013

Less Css and Umbraco

Or in this case more css.  I often run into limitations and annoyances with the limitations of Css rules. But there are solutions for this, namely Less the css preprocessor.  It very simply lets you markup Css files with additional logic and features that css doesn't support.  Then runes a processing function over the file to generate a second rewritten version in raw Css code.  This quickly allows for things like variable,  functions, inheritance, nested rules, and basic math.

Any way this can be implemented a few ways but I crated the simple package for Umbraco to completely automate this process.  It very simply looks for *.less.css files in the normal umbraco Css folder and auto-compiles them on save.  So install this and then simply create a new stylesheet called test.less in Umbraco and you are good to go. Just make sure you always edit the .less version as the compiled file is read only.

Thursday, July 11, 2013

Using a Sophos UTM in Virtual Box

Sophos provides a free home user UTM.  This is a direct update to the older Astaro ASG that I was running.  I recently had to convert over to the new system.  I run it on my FreeBSd server so prefer to use Virual Box to run my VM's.  I ran into a bit of an issue getting the UTM onto the network, but now it's working perfectly.  Here are some simple instructions to get it working.

First of all register and download the VMware x86 or x64 zip file(not the ESX version).  Onew downloaded unzip this file and grab all the *.vmdk files.  The other files can be discarded.

Now go into Virtual box and create a new VM.  Select Linux as  the OS and configure it as desired with the following changes.
  • Add a new IDE disk and browser for and select the base VMDK file. (The one without a -s###.)
  • Add two network cards and change both to use Bridged Networking and change the device driver to Intel PRO/1000 MT Server.
This should be all that is needed to get things working, though I would suggest removing the audio hardware.  Also the UTM shouldn't need much memory, 350-500MB depending on the features you plan to enable.

Now you can boot up the VM.  If you plan to run it as a service you can but for now you need to start it directly in VirtualBox so you can get at the console. Once booted up the system will list and IP to access it at but this won't actually work due to our VM/Nic setup.  So instead go into the UTM's console and logging as root.  The password will be blank and you will need to change it.

Now on the UTM do the following to reconfigure the default network settings. Tab completion will work here.  Also be sure to substitute you settings for the following variables:
$ADDRESS (eg: Internal Ip for the UTM.
$NETMASK (eg: 24) See here for help.
$BROADCAST (eg: Subnet part of Ip filled in with 255 for the rest.
$NETWORK (eg: Subnet of the Ip filled in with 0's.
$INTERFACE Use tab compleation for this and choose witch NIC you want as your internal.








Each set of commands above will print out the new config when the w line is entered so you should be able to identify any errors or problems.  Once finished you can reboot the UTM by typing reboot.

Now if everything is working you should be able to navigate to https://$ADDRESS:4444 and setup the UTM. If you get any error about not being able to change your active connection then one of the settings above must not match or the the itfhw in the last step didn't get set right.

Wednesday, June 12, 2013

Automatic Script/CSS Compression/Rollup in Umbraco MVC Views

Similar to the article about doing this in Master Page Templates but for doing so within MVC Views.

When using Umbraco you may have noticed that the Backoffice Administrative interface uses the Client Dependency Framework to roll up and compress all the different scripts and css resources into two calls.  This leads to a big improvement in responsiveness as the server doesn't need to download as much data nor complete as many requests to render a web page in the admin area.

The good new here is that you can use this same system in your actual site with very little setup.  A few simple step to configure your main site views to use the Client Dependency Framework also.

First of all add the following code to you /Views/Wec.onfig in the /configuration /system.web.webPages.razor /pages /namespaces section.

<add namespace="ClientDependency.Core" /> 
<add namespace="ClientDependency.Core.Mvc" />

Next for all of you sites master templates add the following code someplace in the HEAD block.  Note that you don't want to put this on every templates, just those that are the top level templates.  The golden rule is that every page should call this one and only once.

@Html.Raw(Html.RenderCssHere(new BasicPath("Styles", "/Css")))
@Html.Raw(Html.RenderJsHere(new BasicPath("Scripts", "/Scripts"))))

Now go through all your Views and change your Script and Css includes to the following sample blocks.

  Html.RequiresCss("jquery-ui-1.8.css", "Styles", 0);
  Html.RequiresJs("jquery-1.8.3.js", "Scripts", 0);
  Html.RequiresJs("jquery-ui-1.9.js", "Scripts");

Note that the Priority property is not required.  If set it will load those first in ascending numerical order then load any thing without a priority.  Also the system will detect duplicate inclusions of the same script/css file on the page and only include each one a single time.  You also won't see errors if a script is missing so watch for that as it can be hard to realize if you make a typo.

If you view the site now you will see two lines like this and if you inspect them they will contain all the compressed code.

<link href="/DependencyHandler.axd?s=RandmData&amp;t=Css&amp;cdv=41" type="text/css" rel="stylesheet">
<script src="/DependencyHandler.axd?s=RandmData&amp;t=Javascript&amp;cdv=41" type="text/javascript"></script>

A few other notes  This will not work when compilation/debug is set to True in the web.config.  You will need to disable debug mode to test it and also on the production environment.  Also when the code is compressed it is very hard to debug.  You will want to turn on debug mode on the development system when debugging CSS and Javascript issues or doing development.  The resources will just be loaded with individual script calls in this case.

Also once this is set up there is no reason to use minified scripts in your site anymore.  I suggest replacing them with the full versions for easier code review/editing.

To register dependencies from a Macro Script the code is a bit different.  First add this file to your App_Code directory or a precompiled DLL, then use the sample code below.  Not that this is not needed for Partial View macros.

@using ClientDependency.Core.Mvc
@using JSP.ClientDependency

  .RequireJs("jquery.cookie.js", "Scripts")
  .RequireCss("style.css", "Styles")
  .RequireCss("normalize.css", "styles", 0);