I switched over to FreeBSD a while back and still am in the process of configuring it the way my Ubuntu server used to be set up. One of the things I used to have was uTorrent running, via wine, and auto-loading torrents from a shared directory. This allowed me to queue downloads from anywhere and have them auto-download. It is not possible to set up my 32bit FreeBSD box the same was as ZFS prevents wine from operating. (This is a known issue due to the custom kernel with an increased memory map.)
As such I was looking for a similar solution that would run as a daemon and could automatically process torrent files. And as it turns out the transmission-daemon package does exactly that. Following is a brief review of setting up this service and some customizations I did.
Tuesday, February 23, 2010
Tuesday, February 16, 2010
Convert XML to Array and Array to XML in PHP
This class library provides two XML related functions. One to convert an XML Tree into a PHP array and another to convert a complex PHP object into XML.
The first is the xml_to_array function witch will read through an entire xml tree and convert it into a single multi-dimensional array. That is each node in the root node will be added as items in the array and all children and properties will be added to the respective items. This is done in a completely recursive manner so that nodes within nodes within nodes will be fully read and created as arrays within arrays within arrays. The exact format of the generated array is dependent on the options selected.
The default options will merge all the properties and children directly into the node but this may need to be disabled if you have children with the same name as properties or if you need to distinguish between the two types.
Here is a sample XML file, from a svn dump.
Calling the conversion with the default options would look like so.
Where as the call with no options generates the more detailed but less user friendly output.
A more useful format might be to retail the attribute/children breakdown but processes the rest.
The second function array_to_xml will to the inverse of this and will generate a XML Tree from a complex anonymous array object. This object can contain any amount of data and will be recursively traversed and added to the tree.
Also note that XML Tree's cant have shared nodes or references so if any exist in the source object the data in them will be duplicated in the XML Tree. There is no way to preserve these references with this tool. Also note that no recursion checks are done so an object with circular recursion will lock up the process.
The following array converted to XML will look like so.
Source Code:
The first is the xml_to_array function witch will read through an entire xml tree and convert it into a single multi-dimensional array. That is each node in the root node will be added as items in the array and all children and properties will be added to the respective items. This is done in a completely recursive manner so that nodes within nodes within nodes will be fully read and created as arrays within arrays within arrays. The exact format of the generated array is dependent on the options selected.
The default options will merge all the properties and children directly into the node but this may need to be disabled if you have children with the same name as properties or if you need to distinguish between the two types.
Here is a sample XML file, from a svn dump.
svn log --non-interactive --xml <?xml version="1.0"?> <log> <logentry revision="114"> <author>pynej</author> <date>2010-01-11T14:20:42.200771Z</date> <msg>Removed smarty 2.0 code. </msg> </logentry> <logentry revision="113"> <author>pynej</author> <date>2008-08-08T05:14:31.046815Z</date> <msg>* Changed videos to search recursively. This may be configured by an extra variable in the future.</msg> </logentry> </log>
Calling the conversion with the default options would look like so.
print_r(xmlutils::xml_to_array($xml));
Array
(
[name] => log
[0] => Array
(
[name] => logentry
[revision] => 114
[author] => pynej
[date] => 2010-01-11T14:20:42.200771Z
[msg] => Removed smarty 2.0 code.
)
[1] => Array
(
[name] => logentry
[revision] => 113
[author] => pynej
[date] => 2008-08-08T05:14:31.046815Z
[msg] => * Changed videos to search recursively. This may be configured by an extra variable in the future.
)
)Where as the call with no options generates the more detailed but less user friendly output.
print_r(xmlutils::xml_to_array($xml), 0);
Array
(
[name] => log
[children] => Array
(
[0] => Array
(
[name] => logentry
[attributes] => Array
(
[revision] => 114
)
[children] => Array
(
[0] => Array
(
[name] => author
[value] => pynej
)
[1] => Array
(
[name] => date
[value] => 2010-01-11T14:20:42.200771Z
)
[2] => Array
(
[name] => msg
[value] => Removed smarty 2.0 code.
)
)
)
[1] => Array
(
[name] => logentry
[attributes] => Array
(
[revision] => 113
)
[children] => Array
(
[0] => Array
(
[name] => author
[value] => pynej
)
[1] => Array
(
[name] => date
[value] => 2008-08-08T05:14:31.046815Z
)
[2] => Array
(
[name] => msg
[value] => * Changed videos to search recursively. This may be configured by an extra variable in the future.
)
)
)
)
) A more useful format might be to retail the attribute/children breakdown but processes the rest.
print_r(xmlutils::xml_to_array($xml), xmlutils::XML_MERGE_ATTRIBUTES | xmlutils::XML_VALUE_PAIRS | xmlutils::XML_MERGE_VALUES);
Array
(
[name] => log
[children] => Array
(
[0] => Array
(
[name] => logentry
[revision] => 114
[children] => Array
(
[author] => pynej
[date] => 2010-01-11T14:20:42.200771Z
[msg] => Removed smarty 2.0 code.
)
)
[1] => Array
(
[name] => logentry
[revision] => 113
[children] => Array
(
[author] => pynej
[date] => 2008-08-08T05:14:31.046815Z
[msg] => * Changed videos to search recursively. This may be configured by an extra variable in the future.
)
)
)
) The second function array_to_xml will to the inverse of this and will generate a XML Tree from a complex anonymous array object. This object can contain any amount of data and will be recursively traversed and added to the tree.
Also note that XML Tree's cant have shared nodes or references so if any exist in the source object the data in them will be duplicated in the XML Tree. There is no way to preserve these references with this tool. Also note that no recursion checks are done so an object with circular recursion will lock up the process.
The following array converted to XML will look like so.
print_r($logList);
Array
(
[name] => log
[0] => Array
(
[name] => logentry
[revision] => 114
[author] => pynej
[date] => 2010-01-11T14:20:42.200771Z
[msg] => Removed smarty 2.0 code.
)
[1] => Array
(
[name] => logentry
[revision] => 113
[author] => pynej
[date] => 2008-08-08T05:14:31.046815Z
[msg] => Array
(
[0] => * Added links to the debug section to view the contents of ajax calls.
[1] => * Added query->get_md5 to calculate the md5 hash of a query result.
)
)print_r(xmlutils::array_to_xml($logList));
<?xml version="1.0" encoding="utf-8"?>
<data>
<name>log</name>
<data-item>
<name>logentry</name>
<revision>114</revision>
<author>pynej</author>
<date>2010-01-11T14:20:42.200771Z</date>
<msg>Removed smarty 2.0 code.</msg>
</data-item>
<data-item>
<name>logentry</name>
<revision>113</revision>
<author>pynej</author>
<date>2008-08-08T05:14:31.046815Z</date>
<msg>
<msg-item>* Added links to the debug section to view the contents of ajax calls.</msg-item>
<msg-item>* Added query->get_md5 to calculate the md5 hash of a query result.</msg-item>
</msg>
</data-item>
</data>
Source Code:
xmlutils.php
/**
* This class contains mothods to convert a xml tree into a complex array with nested properties and to convert a complex array object into an xml tree.
*
* @package xml-utils
* @author Jeremy Pyne <jeremy.pyne@gmail.com>
* @license CC:BY/NC/SA http://creativecommons.org/licenses/by-nc-sa/3.0/
* @lastupdate February 16 2010
* @version 1.5
*/
final class xmlutils
{
/**
* Add a levels attributes directly to the levels node instead of into an attributes array.
*
*/
const XML_MERGE_ATTRIBUTES = 1;
/**
* Merge the attribures of a level into the parent level that they belong to.
*
*/
const XML_MERGE_VALUES = 2;
/**
* Add a levels children directly to the levels node instead of into a children array.
*
*/
const XML_MERGE_CHIILDREN = 4;
/**
* Process value as a lone entry under their level and ignore the other attributes and children.
*
*/
const XML_VALUE_PAIRS = 8;
/**
* Split the value of a node into an array on newlines.
*
*/
const XML_SPLIT_VALUES = 16;
/**
* If a value is an array with a single item, just use the item.
*
*/
const XML_SPLIT_SHIFT = 32;
/**
* This function will convert an XML tree into a multi-dimentional array.
*
* @param SimpleXMLElement $xml
* @param bitfield $ops
* @return array
*/
public static function xml_to_array($xml, $ops=63) {
// Store the name of this level.
$level = array();
$level["name"] = $xml->getName();
// Grab the value of this level.
$value = trim((string)$xml);
// If we have a value, process it.
if($value) {
// Split the value into an array on newlines.
if($ops & self::XML_SPLIT_VALUES)
$value = explode("\n", $value);
// If the value is an array with one item, remove the array.
if($ops & self::XML_SPLIT_SHIFT)
if(sizeof($value) == 1)
$value = array_shift($value);
// Store the value of this level.
$level["value"] = $value;
}
// If this level had a value just return the name/value as an array.
if($ops & self::XML_VALUE_PAIRS && array_key_exists("value", $level))
return array($level["name"] => $level["value"]);
// Loop through each atribute of this level.
foreach($xml->attributes() as $attribute) {
// Add each attribure directly to this level in the array.
if($ops & self::XML_MERGE_ATTRIBUTES)
$level[$attribute->getName()] = (string)$attribute;
// Add all the attributes to an attributes array under this level in the array.
else
$level["attributes"][$attribute->getName()] = (string)$attribute;
}
// Loop through each child of this level.
foreach($xml->children() as $children) {
// Get an array of this childs data.
$child = self::xml_to_array($children, $ops);
if($ops & self::XML_MERGE_VALUES) {
// Add each child directly to this level or to the children array of this level in the array.
if(sizeof($child) == 1) {
// If there is only one child then merge it up.
if($ops & self::XML_MERGE_CHIILDREN)
$level[array_shift(array_keys($child))] = $child[array_shift(array_keys($child))];
else
$level["children"][array_shift(array_keys($child))] = $child[array_shift(array_keys($child))];
} elseif(array_key_exists("value", $child)) {
// If there is a value key then merge it up.
if($ops & self::XML_MERGE_CHIILDREN)
$level[$child["name"]] = $child["value"];
else
$level["children"][$child["name"]] = $child["value"];
} elseif(array_key_exists("children", $child)) {
// If there are children, then merge them up.
if($ops & self::XML_MERGE_CHIILDREN)
$level[] = $child;
else
$level["children"][] = $child;
} else {
// Otherwise just assigne yourself.
if($ops & self::XML_MERGE_CHIILDREN)
$level[] = $child;
else
$level["children"][] = $child;
}
} else {
$level["children"][] = $child;
}
}
return $level;
}
/**
* The main function for converting to an XML document.
* Pass in a multi dimensional array and this recrusively loops through and builds up an XML document.
*
* @param array $data
* @param string $rootNodeName - what you want the root node to be - defaultsto data.
* @param SimpleXMLElement $xml - should only be used recursively
* @return string XML
*/
public static function array_to_xml($data, $rootNodeName = 'data', $xml=null, $parentXml=null)
{
// turn off compatibility mode as simple xml throws a wobbly if you don't.
if (ini_get('zend.ze1_compatibility_mode') == 1)
{
ini_set ('zend.ze1_compatibility_mode', 0);
}
//if ($rootNodeName == false) {
// $xml = simplexml_load_string("<s/>");
//}
if ($xml == null)
{
$xml = simplexml_load_string("<?xml version='1.0' encoding='utf-8'?><$rootNodeName />");
}
// loop through the data passed in.
foreach($data as $key => $value)
{
// Create a name for this item based off the attribute name or if this is a item in an array then the parent nodes name.
$nodeName = is_numeric($key) ? $rootNodeName . '-item' : $key;
$nodeName = preg_replace('/[^a-z1-9_-]/i', '', $nodeName);
// If this item is an array then we will be recursine to the logic is more complex.
if (is_array($value)) {
// If this node is part of an array we have to proccess is specialy.
if (is_numeric($key)) {
// Another exception if this is teh root node and is an array. In this case we don't have a parent node to use so we must use the current node and not update the reference.
if($parentXml == null) {
$childXml = $xml->addChild($nodeName);
self::array_to_xml($value, $nodeName, $childXml, $xml);
// If this is a array node then we want to add the item under the parent node instead of out current node. Also we have to update $xml to reflect the change.
} else {
$xml = $parentXml->addChild($nodeName);
self::array_to_xml($value, $nodeName, $xml, $parentXml);
}
} else {
// For a normal attribute node just add it to the parent node.
$childXml = $xml->addChild($nodeName);
self::array_to_xml($value, $nodeName, $childXml, $xml);
}
// If not then it is a simple value and can be directly appended to the XML tree.
} else {
$value = htmlentities($value);
$xml->addChild($nodeName, $value);
}
}
// Pass back as string or simple xml object.
return $xml->asXML();
}
Wednesday, February 10, 2010
Find Duplicate Files in the Terminal
I posted an Automator Service last week for finding duplicate photo's in an iPhoto Library. Here is a slightly modified version of the internal script it uses. You can save this script and run it in a terminal to find duplicate file of any kind in any directory tree of your choice. This can also be included in Automater actions itself with the Shell Script action.
findDuplicates.pl #!/usr/bin/perl # ##################################### # # Filename: findDuplicates.pl # Author: Jeremy Pyne # Licence: CC:BY/NC/SA http://creativecommons.org/licenses/by-nc-sa/3.0/ # Last Update: 02/10/2010 # Version: 1.5 # Requires: perl # Description: # This script will look through a directory of files and find and duplicates. It will then # return a list of any such duplicates it finds. This is done by calculating the md5 checksum # of each file and recording it along with the filename. Then the list is sorted by the checksum # and read in line by line. Any time multiple records in a row share a checksum the file names # are written out to stdout. As a result all empty files will be flagged as duplicates as well. # ##################################### # # Get the path from the command line. Thos could be expanded to provide more granular control. $dir = shift; # Set up the location of the temp files. $file = "/tmp/pictures.txt"; $sort = "/tmp/sorted.txt"; # Find all files in the selected directory and calculate their md5sum. This is by far the longest step. `find "$dir" -type file -print0 | xargs -0 md5 -r > $file`; # Sort the resulting file by the md5sum's. `sort $file > $sort`; open FILE, "<$sort" or die $!; my $newmd5; my $newfile; my $lastmd5; my $lastfile; my $lastprint = 0; # Read each line fromt he file. while() { # Extract the md5sum and the filename. $_ =~ /([^ ]+) (.+)/; $newmd5 = $1; $newfile = $2; # If this is the same checksum as the last file then flag it. if($1 =~ $lastmd5) { # If this is the first duplicate for this checksup then print the first file's name. if(!$lastprint) { print("$lastfile\n"); $lastprint = 1; } # Print the conflicting file's name/ print("$newfile\n"); } else { $lastprint = 0; } # Record the last filename and checksup for future testing. $lastmd5 = $newmd5; $lastfile = $newfile; } close(FILE); # Remove the temp files. unlink($file); unlink($sort);
Tuesday, February 9, 2010
Switch Statment for Smarty 3
Here is the updated {switch} statement for Smarty 3. The new version is NOT backwards compatible but the Smarty 2 version is still maintained here.
11/23/2010 - Updated to version 3.5
Updated to work with Smarty 3.0 release. (Tested on 3.0.5).
I removed the code from this posting, it is now available on github here: https://github.com/pynej/Smarty-Switch-Statement/archives/Smarty-3.0
10/28/2010 - Updated to version 3.3
I have added this project to GitHub at http://github.com/pynej/Smarty-Switch-Statement.
02/25/2010 - Updated to version 3.3
Please note that this update is required for version 3.0b6 or grater. The change is simple renaming the execute methods to compile but it not backwards compatible. The Smarty3.0b5 and below version is still available here.
02/09/2010 - Updated to version 3.2
Fixed a bug when chaining case statements without a break.
02/09/2010 - Updated to version 3.1
Updated the plug-in to once again support the shorthand format, {switch $myvar}. To enable this feature you must add the following line to you code somewhere before the template is executed.
11/23/2010 - Updated to version 3.5
Updated to work with Smarty 3.0 release. (Tested on 3.0.5).
I removed the code from this posting, it is now available on github here: https://github.com/pynej/Smarty-Switch-Statement/archives/Smarty-3.0
10/28/2010 - Updated to version 3.3
I have added this project to GitHub at http://github.com/pynej/Smarty-Switch-Statement.
02/25/2010 - Updated to version 3.3
Please note that this update is required for version 3.0b6 or grater. The change is simple renaming the execute methods to compile but it not backwards compatible. The Smarty3.0b5 and below version is still available here.
02/09/2010 - Updated to version 3.2
Fixed a bug when chaining case statements without a break.
02/09/2010 - Updated to version 3.1
Updated the plug-in to once again support the shorthand format, {switch $myvar}. To enable this feature you must add the following line to you code somewhere before the template is executed.
$smarty->loadPlugin('smarty_compiler_switch');
If you do not add said line the long hand form will still work correctly.sample.php
/**
* Sample usage:
* <code>
* {foreach item=$debugItem from=$debugData}
* // Switch on $debugItem.type
* {switch $debugItem.type}
* {case 1}
* {case "invalid_field"}
* // Case checks for string and numbers.
* {/case}
* {case $postError}
* {case $getError|cat:"_ajax"|lower}
* // Case checks can also use variables and modifiers.
* {break}
* {default}
* // Default case is supported.
* {/switch}
* {/foreach}
* </code>
*
* Note in the above example that the break statements work exactly as expected. Also the switch and default
* tags can take the break attribute. If set they will break automatically before the next case is printed.
*
* Both blocks produce the same switch logic:
* <code>
* {case 1 break}
* Code 1
* {case 2}
* Code 2
* {default break}
* Code 3
* </code>
*
* <code>
* {case 1}
* Code 1
* {break}
* {case 2}
* Code 2
* {default}
* Code 3
* {break}
* </code>
*
* Finally, there is an alternate long hand style for the switch statements that you may need to use in some cases.
*
* <code>
* {switch var=$type}
* {case value="box" break=true}
* {case value="line"}
* {break}
* {default}
* {/switch}
* </code>
*/
Monday, February 8, 2010
Creating Acrobat Digital Signatures with a Root CA for Validation
Recently I was looking into using Adobe PDF Signing. This feature requires that each user have a digital certificate for each user. The problem is that creating the default signatures in Acrobat then every certificate to be imported on every other computer. That is to set up 10 users to all properly authenticate signatures you would have to import 10 signatures onto 10 computers witch becomes prohibitively complex.
There is another option. If the users certificates are all signed with a single CA(Certificate Authority) then only the CA needs to get imported to get all the certificate validation working. This is the approach I used but it is not internally supported by Acrobat and requires a Linux box to create the certificates. This guide will show you how to create a CA and signed digital certificates for your users. Then you simply import the single CA into each computer along with the actual users certificate.
Requirements:
Creating the Certificate Authority:
Run the following command to generate new CA under the current directory. You need to make sure this is in a secure path.
Once done you need to edit the /etc/ssl/openssl.cnf configuration file and update the CA_default.dir variable.
Create an acrobat.cnf configuration for creating user certificates.
Next you probably want to extend the CA expiration date beyond one year. The following command will extend it to ten years.
Finally the cacert.pem to a shared location and rename it to end with a .cer file extension so that the clients can import it. This is the public CA certificate used for validating certificates.
Create a Users Digital Certificate:
Create the new users certificate. You will be prompted to enter the end users password that they will type to sign documents.
Now run the next command to sign the generated certificate with the CA. You will be prompted for the CA password.
Finally run the following command to export this certificate as a PKCS12 package witch Acrobat can import.
You can now copy this file out to the same shared location at the CA. It is password protected and the certificates can be extracted from it in the future so a backup of the generated new*.pem files is not needed.
To extract the certificate and keys you can run the flowing commands.
Import the Certificate Authority and Users Digital Certificate:
On any computers that need to be able to validate signatures all you need to do is import the CA file. To do so simply open up Acrobat and go to Document->Manage Trust Identities. Then browse for the *.cer CA file and import it. After importing you need to select the certificate, select Trust, and check the Use this Certificate as a Trusted Root option.
To enable a user to sign documents on a computers you need to do the following steps. Open up Acrobat and go to Document->Security Settings. Then click Add ID and browse for the proper users *.pfx file. You will need to enter the users password once to install the certificate but users will still need to enter the password when signing documents. These certificates are still password protected so multiple signatures can be loaded onto the same computer without issue.
There is another option. If the users certificates are all signed with a single CA(Certificate Authority) then only the CA needs to get imported to get all the certificate validation working. This is the approach I used but it is not internally supported by Acrobat and requires a Linux box to create the certificates. This guide will show you how to create a CA and signed digital certificates for your users. Then you simply import the single CA into each computer along with the actual users certificate.
Requirements:
- OpenSSL is required to do most of the work.
- Acrobat Reader is all that is required on the user computers.
- One copy of Acrobat Standard is needed to enable digital Digital Rights management on PDF files.
Creating the Certificate Authority:
Run the following command to generate new CA under the current directory. You need to make sure this is in a secure path.
/usr/share/ssl/misc/CA.pl -newcaThe password prompt is the CA password and is needed by the administrator when signing new certificates. The rest of the prompts create the CA identification and signature and can not be changed once set. Once finished the demoCA directory can be moved and renamed as necessary.
Once done you need to edit the /etc/ssl/openssl.cnf configuration file and update the CA_default.dir variable.
[ CA_default ]
dir = /root/keys/CompanyCA
Create an acrobat.cnf configuration for creating user certificates.
echo keyUsage=digitalSignature, dataEncipherment > acrobat.cnf
echo 1.2.840.113583.1.1.10=DER:05:00 >> acrobat.cnf
Next you probably want to extend the CA expiration date beyond one year. The following command will extend it to ten years.
openssl x509 -in cacert.pem -days 3650 -signkey ./private/cakey.pem -out cacert.pem
Finally the cacert.pem to a shared location and rename it to end with a .cer file extension so that the clients can import it. This is the public CA certificate used for validating certificates.
Create a Users Digital Certificate:
Create the new users certificate. You will be prompted to enter the end users password that they will type to sign documents.
/usr/share/ssl/misc/CA.pl -newcert
Now run the next command to sign the generated certificate with the CA. You will be prompted for the CA password.
openssl x509 -in newcert.pem -CA cacert.pem -CAkey private/cakey.pem -CAcreateserial -out newcert.pem -days 3650 -clrext -extfile acrobat.cnf
Finally run the following command to export this certificate as a PKCS12 package witch Acrobat can import.
cat newkey.pem newcert.pem | openssl pkcs12 -export > username.pfx
You can now copy this file out to the same shared location at the CA. It is password protected and the certificates can be extracted from it in the future so a backup of the generated new*.pem files is not needed.
To extract the certificate and keys you can run the flowing commands.
openssl pkcs12 -in username.pfx -nokeys -out newcert.pem
openssl pkcs12 -in username.pfx -nocert -out newkey.pem
Import the Certificate Authority and Users Digital Certificate:
On any computers that need to be able to validate signatures all you need to do is import the CA file. To do so simply open up Acrobat and go to Document->Manage Trust Identities. Then browse for the *.cer CA file and import it. After importing you need to select the certificate, select Trust, and check the Use this Certificate as a Trusted Root option.
To enable a user to sign documents on a computers you need to do the following steps. Open up Acrobat and go to Document->Security Settings. Then click Add ID and browse for the proper users *.pfx file. You will need to enter the users password once to install the certificate but users will still need to enter the password when signing documents. These certificates are still password protected so multiple signatures can be loaded onto the same computer without issue.
Subscribe to:
Posts (Atom)
Project Licenses
These works by Jeremy Pyne are licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported License