Regex to create list of values for SQL statement
In my day I’m currently undertaking a lot of data migrations. One thing I have to do frequently is find a list of records that do and/or don’t exist in one or more tables. This usually involves getting a list of ID’s from a log output text files then doing SQL queries to attempt to find them!
Unfortunately our clients machines are generally Windows based. Thankfully most are kind enough to allow us to use/install Notepad++ (IMO the best notepad software available on Windows).
As such I utilise Regex a lot to find/strip/edit the contents to get the ID’s out and today I had a brainwave I thought I’d share on how to get it out quickly and turn it into a list that you can cut and paste into a WHERE clause of a SQL statement.
In our case the format of the logs are:
<ID><Tab character><Error message in text>
What I want is:
'<ID>',
And I get there with:
s/^(.*)/'\1',/
Or in Notepad++:
Enjoy!
WordPress “Unable to locate WordPress Content directory (wp-content).”
Got stumped on this and a whole bunch of Google results gave a bunch of random fixes.
I personally try and avoid hacking core wp files as future upgrades break my hacks.
Found a plugin that does it here: Third Party Host Fix.
Migrating Magento to new (sub)domain
UPDATE: Well it turns out this fixes the store, but not the downloader/PEAR updater. More work to figure that one out!
I’ve been trialing Magento on and off for a few years and one day I’ll get my butt into gear to migrate my whole store from Zen Cart (which desperately needs to be patched anyway) to Magento, but I’ve been trying to tie it into a new automated pricing script I’ve been developing (also for a year or so).
I’ve also rationalised my count of subdomains on my domain and moved my old magento development environment from dev.domain.com to store.domain.com. Previously migrating domains has been a major PITA for Magento but today I decided to screw around in the database and found it to be rather painless!
The database is in the same location with same username/password so all I needed to do was to update the base_url and here’s all I did:
So a few steps, confirm that the two config_data fields are where you want them:
select * from mag_core_config_data where path like 'web/%/base_url';
I have the following output:
+-----------+---------+----------+-----------------------+---------------------------------+ | config_id | scope | scope_id | path | value | +-----------+---------+----------+-----------------------+---------------------------------+ | 19 | default | 0 | web/unsecure/base_url | http://dev.domain.com/ | | 20 | default | 0 | web/secure/base_url | http://dev.domain.com/ | +-----------+---------+----------+-----------------------+---------------------------------+
So it was a simple case of updating these values:
update mag_core_config_data set value = 'http://store.domain.com/' where path like 'web/%/base_url';
BUT unfortunately it still wasn’t rendering properly. I discovered that the pages get cached! OH NOES!
Oh well, let’s just purge the cache and start again:
#rm -Rf /path/to/magento/install/var/cache
And good to go! Now to upgrade to the latest version.
Finding Duplicates based on Filename
Found this brilliant snippet utilising Perl to find duplicates based on filename only, not comparing size (fdupes wins this!)
Full credit goes to radoulov, a member of unix.com forums and cursory link to post in thread
perl -MFile::Find -e'
$d = shift || die "$0 dir\n";
find {
wanted => sub {
-f and push @{$u{$_}}, $File::Find::name;
}
}, $d;
@{$u{$_}} > 1 and printf "found %s in: \n\n%s\n\n",
$_, join $/, @{$u{$_}} for keys %u;
' <dirname>
vim Search and Replace
Had some grief tonight with a search and replace string.
I am backing up some videos to an external HDD and some are files and some are folders, so I’ve created a rsync script which uses a include.txt file.
rsync script is as follows:
rsync -avr --progress --files-from=include.txt / /media/Ext\ Storage
Then in my include.txt I have the following:
data/videos/dir1/
data/videos/dir2/
data/videos/file1.avi
data/videos/file2.mkv
Which I obtained from doing a find dump into a file find /data/videos/ > include.txt
Except for the includes.txt, it the directories need to have the trailing slash.
I first off tried to create a regex search and replace in vim which found all rows that didn’t have a file extension. I managed to create a expression which DID find them but couldn’t work out how to then wrap that in a NOT.
So I thought screw it, I’ll add a trailing slash / to every row then strip them out and after 45min of screwing around I finally got it.
:1,$ s/\(\.[a-zA-Z]*\)\/$/\1/c
I used /c just to make sure it wasn’t going to go crazy, and it didn’t!
It’s probably not the most elegant or efficient but I have a sense of achievement all the same 🙂
Secure HDD wipe
Selling off an old hard drive, I wanted to wipe it clean.
Found the tip here
shred -vfz -n 100 /dev/hda
Here /dev/hda is my whole hard disk. And I am asking shred to make (-n) 100 passes by overwriting the entire hard disk with (-z) zeros. And shred program (-f) forces the write by changing the permissions wherever necessary.
Setting up a tt-rss daemon on Arch Linux
NOTE: I originally wrote this on the tt-rss.org forums, but thought I’d post it here too.
There doesn’t seem to be support for TT-RSS on Arch Linux, particularly in creating the /etc/rc.d/ daemon. So I hacked together this:
I edited the init.d files for TT-RSS to run on Arch Linux, which uses rc.d and doesn’t have start-stop-daemon. This is the first time I’ve done anything like this so please feel free to 1. provide constructive feedback and 2. be nice
Basically following the instructions as per UpdatingFeeds but use my attached files and place in rc.d instead of init.d
* Extract tt-rss-rc.tar.gz
cp tt-rss.default /etc/default/tt-rss
cp tt-rss.rc-d /etc/rc.d/tt-rss
/bin/chmod +x /etc/rc.d/tt-rss
* If you want Tiny Tiny RSS update daemon to start when you computer start :
Edit /etc/rc.conf
Add to "DAEMONS=(... tt-rss ...)"
* Once all this done, you can use this command line to start the update daemon :
/etc/rc.d/tt-rss start
* And this command line to stop the update daemon :
/etc/rc.d/tt-rss stop
Been working great for me for a couple weeks.
Installing ESXi 4 in a Home Network
NOTE: I actually wrote this back on 16th August but did a Save Draft instead of Publish … DOH!
Background
I’ve run my linux server + Windows VM for a couple years now and the Windows VM does as much as the Linux host does. Coupling this with some issues with VMWare Server 1 on latest kernels and failed attempts to keep VMWare Server 2 stable, I decided to take the plunge into ESXi and virtualise my Linux machine too!
Initial Research
So my initial research indicated my NIC was fine but I can’t use linux software raid, which is how my linux box has been setup! Oh noes, oh well, time to get a hardware RAID controller. The cheapest one on the VMWare HCL is the Adaptec 2405, and found one 2nd hand for around 60% of the cost of a new one.
Problems Problems Problems
1. Virtualising my linux box. The VMWare Standalone Converter can’t convert Linux machines running mdadm/software RAID. BOO. Time to copy my important files off one of my RAID1 array to another single HDD.
2. Can’t initalise the disk. This is the problem and solution: http://ict-freak.nl/2009/03/14/vmware-failed-to-get-disk-partition-information/
3. Accessing the ESXi box via remote SSH:
http://professionalvmware.com/2009/05/26/unsupported-console-and-ssh-on-esxi-4/
4. Can’t access my non-mdadm drive in ESXi. RDM (Raw Device Mapping) is the answer. Here is the solution I used: http://www.daenks.info/2009/07/using-vcb-to-backup-entire-vms/
After solving all these issues though, I have to say, that moving to ESXi was the best thing ever. I’ve only had 1 problem since installing it and that was due to the USB drive becoming corrupted.
That was easily fixed by creating a new USB boot key and copying the local.tar.gz file from the old to new key. Boot up and wallah, good as new! Now I have a backup copy anyway 🙂
linux n00b 
leave a comment