Monday, May 19, 2014

Excessive Disk IO/CPU Wait Time on Sophos

I'v used Sophos(formerly Astaro) for years.  Recently though I'v had some performance issues.  I run it in a VM and the system has been slow to respond to new requests as of late.  Logging into the admin interface was even worse.

Rejoice though for I have found the solution.  If the reporting database becomes corrupted from a power failure or something similar the system will still work as expected but the postgresql process will churn away trying to query the invalid data.  This data isn't particularly important as its just the reporting data, and not even all the reporting data at that.  As far as I can tell its mostly used for the Web Protection reporting.

Anyway its a quick fix to reset the database and fix the high Disk IO issue.  Just log into the console and run the following command to remove and recreate the corrupted data.

sudo /etc/init.d/postgresql rebuild

You can run the following command first to verify that this is an issue.  If you see a high cpu-wa value that is constantly over 95 that you have some data corruption issues.  (Its the second to last column.)

vmstat 5

Shared XBMC running from a Raspberry Pi with NFS and MySQL

Updated 9/10/2015: Additional notes and some unnecessary steps struck out when I rebuilt my Pi with the current Xbian RC2 build.

So I was recently saddened by the death of my first Gen Apple TV.  I had XBMC loaded on there and it would auto-mount my media via NFS shares and connect back to a central database.  I decided to try out the new 35$ Raspberry as an alternate to trying to fix it(it was slow to start up anyway).  I looked at just using a cable to the computer but that was inconvenient.

Result: Perfect Success.  I have had no real issues and it even runs AirPaly now(@TV didn't.)

Hardware

Heres what I bought/collected.  I had most the cables and things.
  • Raspberry Pi ~$40
    • Note that Model B is required for HD quality video.
  • Tiny USB Wifi ~$10
    • Technically you could just use a network cable but the Wifi works flawlessly at 1080p so no need to.
  • HDMI Cable
    • This cable can honestly be as short as you can find as the Pi can just sit right behind the TV.
  • 5V 1A(min) Micro USB Charger
    • Or USB with a USB to Micro USB cable.
    • I use a Apple USB power block(From iPad Air of iPhone 5) and a cheap o USB to Micro USB cable.  But anything that puts out 5V and at least 1 Amp will work.
  • 4GB SC Card
    • They recomend a Class 10 High Speed card for performance, but honestly the 5 year old card that I left in the car, sat on, and broke the lock button off works just fine and boots right up.  (I have has some file corruption issues but that might just be from unplugging it and not the SD card itself.)
  • Some way to program the card
    • A newer iMac or Pc with a SD card reader, or a some sort of SD Card Reader.
  • Borrow a USB Keyboard
    • You will need a keyboard for all of 2 minutes.  You don't need a mouse for anything.  Once you connect to the Wifi you don't need the keyboard anymore.
  • Remote
    • Don't need one.  Use you Smart Phone or tablet as a free fully functional remote.  (See below)

Software

There are a few pieces of software that we can prepare/setup now.

Operating System

First up the Operating System for the Raspberry Pi.  There are a few choices here but I recommend Xbian as it has all the features we need and worked on my first try.  It is also very speedy.  To set it up follow these directions:

  • Grab the SD card and find a computer you can attach it to.  Make sure to get any files off it before you start because this will destroy all the data on it.
  • Go here and download the installer for the computer/operating system you are programming the SD card on.
  • Plug in the SD card and run the installer(as administrator as needed.)
  • Install the latest version of Xbian onto the SD Card.
  • Make sure to safely eject any volumes when done so nothing get's corrupted.

Remote Software

While that is installing you can get the Remote Control software for your smart phone/tablet to control the Raspberry Pi.
  • MPEG2 Licence ~$5
    • You will need this later to play YouTube, some older types of content, and to receive AirPlay video content. (Though rented movies won't work due to encryption.)
  • iOS - Free
  • Android - Free
  • Other
    • There is a web interface you can use from other devices to control the Raspberry Pi but I don't recommend that.  Instructions for that are included below in the configuration.

Setup

Setup is a cinch.  Plug the memory card in. (Make sure it unlocked)  Plug in the Wifi Card and temporary keyboard.  Hook up the HDMI to your TV and change the TV to the proper input.  Then finally plugin in the power and watch the magic happen.

Configuration

Note: I had a few problems during setup.  When I would reboot form time to time the system would not come back up.  I ended up having to unplug the Pi and stick the memory card back into my desktop and mount/eject it to get it to start up again.  Not that it's working i don't unplug it so I'm not sure if this is still a problem.  I suspect it is due to the poor quality of my SD card.

Networking

When booting up your Pi for the first time you will need to configure the network. In the setup wizard go to the network tab and switch to the WLAN interface(wlan0).  Then just switch to DHCP Mode and join your wireless network.  You can set a static IP here but nothing we will use it for will require a static IP and you can access it by the Host Name (xbian.local) as needed.

Note: If the keyboard isn't work try to unplug and plug the keyboard back in.

Now make sure you can connect to the Pi from the network.  On your computer use SSH to connect.
  • On OSX open the Terminal and type in the following command and press enter.
  • ssh xbian@xbian.local
  • On Windows you will need to download Putty and fill in xbian for the usernam, xbian.local for the host name, and select SSH for the type.
In either case you should get prompted for a password which is raspberry.

If you have problems go into Settings > Xbian > Networking and make sure thet you successfully joined the network and have an IP address.

Remote Control

We want to get this working next as it will let us use the phone instead of needing the keyboard.
  • In XBMC go into Settings > Services > Web Server and check Allow control of XBMC via HTTP. (On by default now.)
  • Also go into Settings > Services > Zeroconfig and check Announce these services to other systems via Zeroconf.
  • Then on your phone open up the remote control and add a new host.  On an iPhone you can just click Find XBMC and the Pi will show and and configure itself automatically.
  • You may need to manually configure the server on Android devices.
Once the remote is working we won't need the keyboard any more, you can even use the keyboard on your phone to fill in text fields.  See the remote app help for ore details.

Shared XBMC Database

If you wish to use a central database to run XBMC from multiple computers then install MySQL on the file server and create two database's for XBMC.  Then on the Pi you can edit your config to connect to the shared database.  This is not required to use NFS, but rather if you want to use the same meta-data(playes, details, pause point) from multiple systems.

To enter the database information first SSH into the Pi and then run the following.
sudo nano /home/xbian/.xbmc/userdata/advancedsettings.xml
Then inside of the <advancedsettings> add the following filling in the correct values for the host, database, and login details and save the file.
<videodatabase>
  <type>mysql</type>
  <host>192.168.1.10</host>
  <name>xbmc_video</name>
  <user>xbmc</user>
  <pass>xbmc</pass>
</videodatabase>

<musicdatabase>
  <type>mysql</type>
  <host>192.168.1.10</host>
  <name>xbmc_music</name>
  <user>xbmc</user>
  <pass>xbmc</pass>
</musicdatabase>

Shared Media with NFS

Now we want to get connected to a central data store.  Personally I run a FreeBSD/ZFS system with Raid5 but any NAS type setup will work.  The performance of Samba/Windows file Sharing is not up to snuff though so we need to use NFS instead.  Look into configuring NFS exports for your operating system and export the media directories.

Now to configure the Pi we can use the XBMC interface but I have had limited success with that option.  Instead I would recommend setting up the mounts in Xbian directly and then adding the media as a local directory in XBMC.

First connect to the Pi in SSH.  Then do the following steps updating/adding mount points as necessary:

Change XBMC to delay starting until the network is fully activated.
sudo nano /etc/init/xbmc-preload.conf
Replace the following line with the next one and save the file.
start on started mount all
start on stopping mountall and started networking and net-device-up wlan0


Enable statd for remote film locks (In Xian RC2).
sudo nano /etc/default/nfs-common
Change the NEED_STATD line to:
NEED_STATD=yes

Manually mount each share to make sure they are working.
For each share you want to connect to from the Pi type in the following and update the directory name, server address, and full server path as applicable.
sudo mkdir /media/video
sudo mount -t nfs 192.168.1.10:/usr/local/srv/video /media/video
ls /media/video

Assuming that worked set them to auto-mount at system boot.  Don't worry, the system won't hang if they are not available instead just showing empty directories.
sudo nano /etc/fstab
Add the following line to the end of the file for each share and adjust the directory name, server address, and full server path as applicable and save.
192.168.1.10:/usr/local/srv/video /media/video nfs defaults 0 0

Finally we need to make XBMC aware of this new content.  You can do this in XBMC but I find it easier to just update the config file directly.
sudo nano /home/xbian/.xbmc/userdata/sources.xml
Add the following section inside of the <sources> tag and save.  Make sure you don't duplicate the <video> tag if there is already one present but update it to match instead.
<video>
  <default pathversion="1"></default>
  <source>
    <name>Shared Media</name>
    <path pathversion="1">/media/</path>
  </source>
</video>

Note: If you using a shared database and multiple devices then your sources configuration needs to match.  You may need to crete some symlinks to get the media to show up in the same place on all the computers. 

Reboot

We are to a point now where you can reboot and everything should restart and come up correctly.  You should now be able to play moves and update your library.  If you didn't connect to a existing library you will need to run a full scan which will take some time.

You should be able to restart safely by resting the power but I recommend logging into SSH and run sudo restart the first time just to be safe.

MPEG2 Video

To allow for decoding of some older formats, various internet streams, and AirPlay Video you will need to do the following to get a MPEG-2 License. 

  • SSH into the Pi and run the following command. 
  • cat /proc/cpuinfo | grep Serial
  • Get the serial code from the output and go here to buy a license.
  • In XBMC go to Settings > Xbian > Codecs and enter the code you get from your purchase.  Remember you can use the keyboard on your phone instead of navigating the onscreen keyboard.

Other XBMC Settings

  • Enable AirPlay playback.
    • Settings > Services > AirPlay >
      • Allow XBMC to Receive AirPlay Content: Yes
      • Use Password Protection: No
  • Stack related movies into sets.
    • Settings > Video > Library > Group Movies in Sets: Yes
  • Enable Surround Sound if you have it and DTS and AC3 if your receiver supports them.
    • Setting > System > Audio Output
      • Speaker Configuration
      • DTS Capable Receiver
      • AC3 Capable Receiver

AirPlay Video

To get AirPlay to work constantly you will need to disable IPv6 on xbian.  It is enabled by default so to turn it of logo in via ssh and run the following command.
sudo nano /etc/sysctl.d/disableipv6.conf
Then press add the following line and save the file.
net.ipv6.conf.all.disable_ipv6=1

Also you need to make sure that you didn't set a username/password for AirPlay as that won't work.  If so navigate to Settings > Services > AirPlay and uncheck Use Password Protection.

This is still very glitchy/unreliable for me.

Results

Things that are confirmed working in this setup:
  • H264 up to 1080p
  • MPEG4(divx, xvid, etc) up to 1080p
  • DTS 5.1
  • AC3 5.1
  • Stereo
  • Volume control via Remote App
  • NFS Playback @ 1080p
  • Subtitles (hard and soft)
  • MKV containers up to 1080P
Does not work:
  • There are, sadly, no click sounds when navigating the UI.
  • Some configurations in MKV files that won't even start playing and just come back with a File Unavailable error. Will investigate further
  • FLAC Audio. When do I even have this?

Notes/Help

Extra scrapper rules

I have also updated my creepers to support the following folder structure:
  • /Show Name/Season 1/01 Title.mp4
  • /Show Name/Season 1/Show Name 01 Episode Title.mp4
  • /Show Name/Season 1/Show Name 01.mp4
  • /Show Name/Season 1/Show Name S01E01 Episode Title.mp4
  • /Show Name/Season 1/Show Name S02E01.mp4

To do this i have to edit run the following and add the next line.
sudo vi /home/xbian/.xbmc/userdata/advancedsettings.xml
Then inside of the <advancedsettings> add the following section and save the file.
<tvshowmatching>
    <regexp>[/\\]Season[\._ ]([0-9]+)[^\\/]*[\\/][^\\/0-9]*[\._ ](?:S?[0-9]+[xE])?([0-9]+)([^\\/]*)</regexp>
</tvshowmatching>

Issues with updates

You may need to run the following to get updates working.
apt-get -o Dpkg::Options::="--force-all" upgrade

Offline Backup

You may also want to create a binary backup of the SD Card once configured to quickly restore from if you have any file system corruption issues.  This has bit me more then once so I would recommend it.  Its a quick process.  

Just plug the memory card back into a computer and make a image of the SC card.  On OS X you can do this quickly using Disk Utility.  Select the entire drive(not just the first partition) and click new image to save copy of the entire SD card.  You can later restore the image using the Restore tab.