Counting occurrence of a particular word in Vim

January 28th, 2009

There is no built in word count feature in Vim, but a simple use of the substitute command can be used for the same purpose.

:%s/word//gn

Vim returns something along the lines of:

14 matches on 47 lines

Basically we are performing a search and replace but telling Vim not to take any action. If we break down the command we can see how it works.

:s – is the substitute command (find and replace)

% – putting the ‘%’ in front tells Vim to substitute across the whole file (not just this line)

<word> – this is the word we are counting (it could also be a regex expression, see below)

g – the ‘g’ switch tells Vim to search the whole line (globally) and not stop after the first match

n – the ‘n’ switch tells Vim to take ‘N’o action. This is what we want as we are only counting.

Using  regex:

Usually this command counts all occurrences of the word, but we can make it more useful with a sprinkling of regex. Say you want to count all lines that begin with comments (shell style):

:%s/^#//gn

Or all lines that are empty or only contain whitespace:

:%s/^\s*$//gn

The source of this tip is from the tips built into the Vim online help. If you enter the following command, you will get the help page version of what I have documented here (took me a while to track this down inside Vim).

:h count-items

Fix for Marvell SATA/IDE controller on Debian

October 23rd, 2008

I recently purchased an ASUS P5Q-PRO motherboard to use as a basis for an upgrade to my home server.  Unfortunately I had not done my research and ran into some issues with installing and running Linux.  Most of these issues revolved around the fact that ASUS had implemented IDE using the Marvell 88SE6111 controller.

Installation issues

Initially, booting up off the Debian netinst CD, everything seemed fine, but once I got to the CD drive detection stage the installer threw an error: “No common CD-ROM drive was detected“.  I tried manually with the CD drivers on offer from the installer but had no luck.

Whilst I was a little concerned for the future of this motherboard, I thought that once it was up a running there would probably be a fix available somewhere for this problem.  So to get to that point I decided to install Debian from a flash disk – this worked fine (there are plenty of guides available to show you how to do this, or you can check out my previous post Installing Debian from a USB Flash Disk).

Still no IDE

I now had a new Debian Lenny system up and running, but unfortunately my only optical drive (being IDE) was not working at all.  I was aware that the module I needed for this controller was pata_marvell and performing the following command confirmed it was already loaded.

lsmod | grep marvell

Solution

In the end the solution to this issue was quite simple.  Whilst I had the right module installed and loaded, the order in which it was loading was the issue.  The fix was to have the pata_marvell module load up before the ahci module.  To do this we need to perform two small tasks (as root).

First we need to edit:

/etc/initramfstools.d/modules

To the bottom of this file add the following lines:

pata_marvell
ahci

With this file updated, we need to update our initramfs, this can be done with the following command:

update-initramfs -k all -u

This will update all the kernel options you currently have on your system.  With that done, reboot and check if you have your IDE devices available.  My DVD drive showed up as /dev/scd0.

Whilst I couldn’t find this complete solution in one place, the combination of the following websites allowed me to piece it together:

http://fixunix.com/debian/542307-bug-493657-workaround-2-6-25-2-6-26-marvell-88se6121-88se6145-pata_marvell-ahci-regression.html
http://www.gossamer-threads.com/lists/linux/kernel/968813
http://www.linuxquestions.org/questions/linux-general-1/module-load-order.-648599/

Using Vi commands to control your shell (ksh, Bash)

September 13th, 2008

Recently I was forced to use the Korn Shell (or ksh) when I had to perform some minor tasks on an AIX server. So of course I just jumped in and started using the shell. A few seconds later I was thoroughly frustrated as I discovered I had no command history and couldn’t use backspace or command completion (as you can probably guess I’m a Bash user).

The Vi mindset

So begrudgingly I resorted to a few web pages to work out what was going on. To my pleasant surprise, all I had to do was switch my thinking from Bash to Vi. I’m a big Vi(m) fan and feel quite at home using Vi like commands so I was quite happy to discover this was all I had to do. Command history: just hit ESC and use the Vi “up” which is “k” to go back in history, or Vi “down” which is “j” to go forward in history – easy.

Keeping in this mindset I began investigating other commands. Jumping back in history, then a quick “cw” to change a word, “A” to append to the end of that command. I wont bore all the people that don’t use Vi(m) with a list of commands, because if you don’t, then all this probably sounds like a pretty painful way of using a shell. But if you do use Vi(m) regularly, you quickly learn that a few carefully selected keystrokes can save you many repetitive backspaces or left arrows etc.

Escape from home

The other great thing about using Vi(m), or a Vi like interface, is that your fingers never need to leave the home keys. The only exception to this is that pesky ESC key that has been banished to the far corner of the keyboard. But never fear, there are some fixes to this.

The answer is to remap your CapsLock key to be another Ctrl key. Not only does this open up a wealth of shortcuts that can be performed without taking you fingers away from the home keys, but it also allows you to type an ESC equivalent, which is Ctrl-[.

If you are running X11 you can do it in the (re)configuration. Running Gnome you can do it through keyboard preferences, I’m guessing there is something similar in KDE and other window managers. You can even do it in Windows with a simple registry hack.

More information for Linux, Mac and Windows at the below links:

http://www.manicai.net/comp/swap-caps-ctrl.html

http://johnhaller.com/jh/useful_stuff/disable_caps_lock/

Dessert time

Anyway, whilst I was pretty impressed with the interface that ksh had on offer (once I understood it), I wasn’t totally sold. Command line completion was there (ESC+\) but it was nowhere near the level of Bash’s usefulness.

But…..inspired by this discovery, it didn’t take long to work out that I could have my cake and eat it too! A simple command in Bash allows these Vi like commands to be used:

set -o vi

This way you can keep on using Bash as you would, but hit the ESC key (or Ctrl-[) and you have the power of Vi at your fingertips!

Installing Debian from a USB Flash Disk

May 4th, 2008

It was time for a fresh install of Debian on my ThinkPad X31. After 3 years of filling it up with random programs and university related stuff I felt this was the easiest path to a nice clean, fast system. I was also at a point where I felt comfortable in totally obliterating my Windows partition (which I never used) which had been sitting there as a kind of security blanket.

Anyway, for those that don’t know, the X series of ThinkPads are the ultra-portable models and in most cases come with no optical drive (unless you have the docking station). Previously I had used a USB based DVD drive to install the OS’s on it, but that was currently in use on another system (and installing from a flash disk did seem like a slightly nicer solution).

Getting Started

All we really need to get going is a flash disk (anything from 256MB and up should do), a network connection and an existing Linux system. As I was using the netinst ISO (160MB) most of the OS will be downloaded over the network after the initial system has been installed. The other point to note is (because we are using the easy way), everything on the flash disk will be wiped so double check it and backup if necessary.

Getting the files

There are only two files that we need, both can be found from http://www.debian.org/distrib/netinst.

  • debian-XXXX-i386-netinst.iso
    This can be found by clicking on your architecture type (i386 for most) in the Small CDs section (replace XXXX with the current version).
  • boot.img.gz
    This can be found by clicking on your architecture type in the Tiny CDs, floppy disks, USB sticks, etc section. You then need to navigate to the hd-media folder.

Setup the flash disk

So now we need to setup our flash disk (remember, EVERYTHING WILL BE WIPED!!). Insert the flash disk and determine its location. The easiest way to do this is to execute ls /dev/sd* before and after you insert the disk and check what drive appears, or alternatively, a few seconds after plugging in, execute dmesg | tail. For me it was /dev/sdb. Now we need to put the boot image on the disk, this is done by executing:

zcat boot.img.gz > /dev/sdb

(zcat is a nice little program that rolls gzip and cat into one)
Once that’s done, we need to copy the ISO over to the disk as well. You might need to mount the disk if it isn’t done for you automatically. This can be done by executing:

mkdir /mnt/sdb
mount /dev/sdb /mnt/sdb

and then

cp debian-40r3-i386-netinst.iso /mnt/sdb

(Make sure you adjust for your version, architecture and disk location)

Boot Away

We are now all set to do the install. Just plug in your flash disk before turning on, and select it as the boot device when booting up. Make sure you have your network cable plugged in so you can get more than a very basic system installed. You now also have a convenient rescue and recovery tool.

Customizing the Bash prompt

April 14th, 2008

Customizing the Bash prompt is, in essence, a relatively easy task. All we need to do is modify the PS1 environment variable. To have a look at your current prompt string, execute the following in a Bash shell:

echo $PS1

Depending on you distribution, this may be a relatively simple output or it may be full of special escape codes to set the colors and do other interesting things.

By adding special characters in our PS1 we can change the elements that make up the prompt, their order, their color and their layout. For example, we can use the following command to change our prompt temporarily:

PS1=”\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$”

On a black background this will look something like this:

Example prompt

The “\u”, “\h” and “\w” are special elements that get replaced with the current username, hostname and working directory respectively. The numeric elements such as “\[\033[01;32m\]” are used to set either the foreground or background color.

Now at this stage, most guides on the Bash prompt will give you a list of all the possible escape codes and color codes you can use and send you packing. This is fine if you know exactly what you want but if you want to try out different combinations it can get a bit tedious. It was this situation that motivated me to write a program to ease the process of creating and trying different prompts.

impromptu

impromptu is a simple program I have written in Perl/GTK to make it easy to create and manage prompts for the Bash shell. It allows you to easily try out different combinations with different colors and saves the user from needing to know all the escape codes and color codes. Prompts can be saved and loaded, automatically installed and un-installed and previewed on the fly.

Here are a couple of screenshots:

Screenshot 1

Screenshot 2

impromptu can be downloaded from here.

Whilst fully functional, it is a young program, so let me know if you have any suggestions or if you find any bugs.

Contact: impromptu@omobos.com.

Using “dd” and “/dev/urandom” to write random data to a disk

April 6th, 2008

dd is an essential Linux tool, kind of a Swiss Army knife. That said, it took me a couple of years on Linux before I even used it once. Whilst there are plenty of uses for dd, today we are using it to fill a hard disk with random data.

There are two possible reasons for doing this: wiping confidential data from a hard disk prior to disposal or increasing the level of protection given by using encrypted disks.

Wiping

Filling a hard disk with random data will prevent all but the most determined people (with specialized tools) from recovering the data. That said, it is probably better done with tools built specifically for the job like shred.

Raising the effectiveness of encryption

Filling a hard disk with random data prior to storing encrypted data on it raises the level of security significantly, making it much harder to crack the pass phrase or key. I’m not a cryptographer and don’t fully understand the process, but I believe it’s something like this.

If a brand new hard disk, or a non-encrypted disk is used in an encrypted setup, there will be parts of the disk that have recognizable data (or completely blank spots) on them. By analyzing the disk, it would be possible to find an edge between encrypted data and non-encrypted data. This information can then be used to aid in the attack against the encryption.

By filling the disk with random data first, it becomes near impossible to spot the edge between the encrypted data (which looks random) and the unused parts of the disk.

Generating Random Data

On a Linux machine we have two main sources of random data: /dev/random and /dev/urandom. /dev/random provides a very high quality random output that is generated using environment noise from device drivers such as sound cards or mouse movement. You can see this by executing:

cat /dev/random

and then moving your mouse around. When you first execute the above command, there may be a fair amount of random data stored up, but once it is exhausted it only comes through very slowly. This means that to fill up a hard disk with data from this source could take years. Enter /dev/urandom.

/dev/urandom (or unlimited random) is, as the name suggests an unlimited source of random data. The quality of the randomness is not as high as that of /dev/random as it uses both input from /dev/random as well as pseudo random generation algorithms. That said it is more than random enough for our purpose.

Putting it together

So to fill up our disk (/dev/sda is this example) with random data involves just one command (to be run as root). The time command at the start is optional but will return the time taken when the command completes.

time dd if=/dev/urandom of=/dev/sda

Now, depending on your disk size and processor speed, this could take from a couple of hours up to a couple of days. As we are using the pseudo random generation algorithm, a side effect is that your CPU will be running up near 100%. I recently used this command to fill up a 400GB disk on a PC with an Athlon-64 3000 processor. The output is shown below.

time dd if=/dev/urandom of=/dev/sdb
dd: writing to `/dev/sdb’: No space left on device
781422769+0 records in
781422768+0 records out
400088457216 bytes (400 GB) copied, 141572 seconds, 2.8 MB/s
real 2359m32.492s
user 4m8.912s
sys 1781m26.368s

A quick bit of maths tells us that it took a bit under 40 hours to complete.

Checking on progress

The dd command doesn’t give us any useful feedback whilst it’s running, but we can force it to cough up some useful data by prodding it with a USR1 signal. Assuming you only have one instance of dd running we can use:

kill -USR1 `pidof dd`

(Note, they are backticks around the pidof dd command.)

So sit back, make yourself a cup of coffee (or 40) and let your system struggle under the responsibilities of generating a continuous stream of pseudo-random data.

(In the near future I will be placing a full guide about using encrypted hard disks in the guides section of omobos.com)

Convert an image to XPM format suitable for embedding in Perl

March 24th, 2008

Recently I was working on a single file GTK/Perl program which used an external image file for a logo. I didn’t want to distribute the image file along side the actual program as this was a little messy and probably not worth it for just one logo. So before doing away with the logo, I investigated if there were any possible ways to embed the image data in the Perl code.

XPM

It seems that the defacto standard for representing image data in ASCII format is XPM or X PixMap. This is usually represented in a way that is compatible with programs written in C (the XPM2 format does away with any language specific details).

Getting the XPM data

Getting the image data in XPM format is relatively easy if you have the ImageMagick suite of tools installed. If so, then just run the following:

convert inputfile outputfile.xpm

The trick here is to make sure your output file ends with a .xpm extension. ImageMagick will recognize this and do the conversion to XPM for you.

Converting to Perl code

As you might have noticed if you looked at the resulting XPM file, it is in standard C notation. Not very useful for a Perl program. Never fear, a sprinkling of substitution and regex magic will bring us home.

“s/\’/\\\’/g”
s/\”/\’/g
“s|^/\*.*\n$||”
“s/^static.*$/my \@xpm_image = (/”
“s/^};$/);/”

In order from top to bottom we have:

  • Escape all the single quotes
  • Convert double quotes to single quotes
  • Remove lines that start with a C-style comment
  • Reformat the variable declaration for Perl
  • Fix up the closing brace on the array definition.

With all these substitutions done, we can embed the resulting code in a Perl program and use it with functions that expect XPM data, Gtk2::Gdk::Pixbuf->new_from_xpm_data() for example.

All together now…….

Ok, lets put it all together into a Bash script. As with any shell script, we need the shebang on the first line:

#!/bin/bash

then we have filename definitions:

inputfile=$1 # $1 is the first argument passed to the script
xpmfile=$inputfile.xpm
perlfile=$xpmfile.pl

ImageMagick conversion and rename:

convert $inputfile $xpmfile
mv $xpmfile $perlfile

and the substitutions (the “i” argument means substitutions are done in place, i.e. there is no separate output file)

perl -pi -w -e “s/\’/\\\’/g” $perlfile # All single quotes get escaped
perl -pi -w -e s/\”/\’/g $perlfile # Double quotes -> single quotes
perl -pi -w -e “s|^/\*.*\n$||” $perlfile # Remove c-style comment lines
perl -pi -w -e “s/^static.*$/my \@xpm_image = (/” $perlfile # Change variable declaration
perl -pi -w -e “s/^};$/);/” $perlfile # Change closing brace of array

If you want to download or view the script in it entirety, have a look at img2xpm_perl at omobos.com.

Color coded man pages with Vim

March 23rd, 2008

Want a color coded man page viewer with hyperlink support? Well look no further. This simple script is a wrapper around the built in :Man function in Vim. Basically it allows Vim to be started up in man mode and adds some key shortcuts like the default man page viewer.

Have a look at this screenshot, it shows the color coding of various elements, hyperlinks for other commands and also shows the folding feature.

Using vim for man page viewing

Enabling the man plugin

Before we begin, we need to make sure the filetype plugin for man is enabled in Vim. To do this we need to source $VIMRUNTIME/ftplugin/man.vim from our .vimrc file.

I have the following line in mine:

source /usr/share/vim/vimcurrent/ftplugin/man.vim

Your location may vary depending on versions/distributions etc.

Using the :Man function

Once we have enabled the plugin, we can view man pages in Vim using the :Man command (i.e. :Man vim). We can now also use the below command in place of our standard man pager.

vim -R -c “:set foldmethod=indent” -c “:set foldnestmax=2″ -c “:set foldenable” -c “:Man $1″ -c “:normal ” -c “:q” -c “:set nomodifiable” -c “:map q :q<CR>” -c “:map <space> “

Place this command inside a shell script, put it on your path and/or set up an alias to it like follows:

alias man=”/home/user/scripts/vimman.sh”

Note: the above command has some non-printing characters in it, such as “Control-W” and “Page-down”. As such, just copying the text from a web browser will probably not work. You can checkout the full script vimman.sh at omobos.com

Please explain…..

The above command is pretty simple if we break it down a little. To start with the ‘-R’ option puts us in read-only mode. The rest of the arguments are actually Vim commands, this is indicated with the “-c” option. The first three relate to folding and can be removed if you prefer not to have folding enabled. Following that we have our :Man command which opens the man page in a new split window. To remove the split window, we jump to the other window (“Control-W. Control-W” – not printed correctly above) and then execute “:q” to close it. The last two commands set up some keyboard shortcuts. Pressing “q” quits immediately and pressing “Space” gives us a “Page-Down”.

Hyperlinks

When viewing man pages in Vim, other referenced commands are highlighted (in blue usually). By placing the cursor on the command name and pressing Ctrl-] you will jump to the man page of that command. Ctrl-T will jump back to the original page.

Finding running processes

March 23rd, 2008

Here is a simple alias to help track down running processes and their PID’s.

alias fproc=’ps aux | grep ‘

This uses the ps command to list running processes and then filters the result with grep using the keyword you specify. For example:

user@machine:~$ fproc firef
user 3413 2.9 12.8 256936 127640 ? Rl Mar23 14:38 /usr/lib/iceweasel/firefox-bin -a firefox
user 31952 0.0 0.0 3860 716 pts/2 R+ 01:16 0:00 grep –color firef

(Note: the grep command will always show up in the output as it too has the keyword in the process name.)

For more information on aliases checkout my previous post or have a look at my custom .bash_aliases file at omobos.com

Bash aliases

March 23rd, 2008

The Bourne again shell (Bash) has a useful feature called aliases.

Aliases allow common commands (along with arguments) to be executed with an alternate command name. For instance using this command;

alias mv=”mv -iv”

means that whenever I use the mv command, the -iv argument will be automatically added resulting in interactive mode (prompting before overwriting) and verbose output. For example:

~$ touch test01.dat test02.dat
~$ mv test01.dat test02.dat
mv: overwrite `test02.dat’? y (Interactive mode)
`test01.dat’ -> `test02.dat’ (Verbose output)
~$

Many commands can have default behavior added to them by adding your most commonly used arguments. Some other examples:

alias grep=’grep –color’ # Highlight search word
alias df=’df -h’ # Display in human readable format
alias du=’du -h’ # i.e. 41G, 217M

But what if you use the ls command a lot but also in the form ls -la? To get around this we can introduce new commands (just make sure you don’t mask over existing commands). For example, I have the following aliases for the ls command.

alias ls=’ls –color=auto’ # Use colors
alias l=’ls -l’ # Long format, username, size etc.
alias la=’ls -la’ # Long format with hidden files
alias lsd=’ls */ -d’ # Only display directories

Making things permanent

When the alias command is used in a shell, the changes are only local to that shell and will be lost when it is exited (exit or Ctrl-D). To make changes permanent, we need to add the aliases to the bashrc file (~/.bashrc for the current user, or /etc/bash.bashrc to make the changes system wide). Similar config files exist for other shells.

But we can take this one step further and keep everything nice and organized. If we keep all the aliases in a separate file (~/.bash_aliases), we can source them from our shell config file and make them a lot easier to maintain. Most likely you already have the following lines in your bashrc file:

# Alias definitions.
# You may want to put all your additions into a separate file like
# ~/.bash_aliases, instead of adding them here directly.
# See /usr/share/doc/bash-doc/examples in the bash-doc package.#if [ -f ~/.bash_aliases ]; then
# . ~/.bash_aliases
#fi

If so, just uncomment those last three lines and keep your aliases in ~/.bash_aliases.

You can check out my complete alias file at omobos.com/data/scripts/.bash_aliases