Geek on the road

open source, open knowledge

Apache Redirect HTTPS

| Comments

If you need to set up redirection with Apache, do not waste your time using the Rewrite module, instead use Redirect, the easy way:

1
2
3
4
5
6
7
8
9

<VirtualHost *:80>
    ServerName lab.sgmac.com
    Redirect / https://lab.sgmac.com/
</VirtualHost >

<VirtualHost *:443>
    ServerName lab.sgmac.com
    ... 
</VirtualHost >

I find this pretty straightforward and more readable.

Rewriting Git Commits

| Comments

In this post I’ll explain how to rewrite your git commits and how to upload your private git repositories to github, but keeping your commit history. Recently I forgot to define my email address on the ‘.gitconfig’ file, this means all my history is dirty, and I wanted to fix it. Of course you can rewrite your commit history.

These are my last two commits on a repository with the wrong email address,

1
2
3
4
5
6
7
8
9
10
11

commit 2c096fc8c8fa425bfd6c11a1223ec9a512383784
Author: Srgio Galvan <false_email@gmail.com>
Date:   Thu Jun 28 15:40:08 2012 +0300

    Add filter example

commit b70b8832df12a53069ae36e928ffb7fbce2dea2e
Author: Sergio Galvan <false_email@gmail.com>
Date:   Wed Jun 27 00:22:18 2012 +0300

    Last exercise

Then I found git filter-branch(1) :

1

git filter-branch --env-filter ' export GIT_AUTHOR_EMAIL=good_email_address@gmail.com'

It’s pretty straighforward, let’s check the new email address:

1
2
3
4
5
6
7
8
9
10
11

commit 4cf11c1d39fd4d91792d37f90ca51eb5b9037477
Author: Sergio Galvan <good_email_addres@gmail.com>
Date:   Thu Jun 28 15:40:08 2012 +0300

    Add filter example

commit f391ad1d131b287df8aa7df15e01ce707b64419f
Author: Sergio Galvan <good_email_addressgmail.com>
Date:   Wed Jun 27 00:22:18 2012 +0300

    Last exercise

As you might already know, the SHA-1 hash is different, now you can’t keep pushing to your remote if you ever had one. On the other hand, I have my respository almost ready to pushing into github. First create your respository in github, and the proceed with the next step:

1

git  push --mirror git@github.com:githubuser/myrepo.git

There are more properties you might want to define, check theme out.

Puppet Apt-sources

| Comments

I started making my own puppet modules and I want to share apt-sources. It’s a pretty straightforward module to manage debian repositories in a simple way. If you’d like getting started with puppet keep reading.

Basically the module allows to define one or more repositories, besides of the area. In Debian we find three areas, main,contrib and non-free, though the last two are not part of Debian. In any case, you could find different areas when using non-official repositories. The module itself only updates the /etc/apt/sources.list file, you do not forget to update your system afterwards.

init.pp 
1
2
3
4
5
6
7
8
9
10
11
12
13

 class apt-sources (
        $release = $::lsbdistcodename,
        $area = undef,
        $sources = undef
){
    file { '/etc/apt/sources.list':
            ensure  => present,
            owner   => 'root',
            group   => 'root',
            mode    => '0644',
            content => template('apt-sources/sources.list.erb');
    }
}

And now we only need to define the template.

sources.list.erb 
1
2
3
4
5
6
7
8
9
10
11
12
13

<% @area = area.is_a?(Array) ? area.flatten.join(" ") : area  %>

# Default repositories
deb http://ftp.ie.debian.org/debian/ <%= release %> main
deb-src http://ftp.ie.debian.org/debian/ <%= release %> main

<% if sources.is_a?(Array) %>
<% sources.each do |repo| -%>
deb <%= repo %> <%= release %> <%= @area %>
<% end -%>
<% else %>
deb <%= sources %> <%= release %> <%= @area %>
<% end %>

And some examples:

sources.pp 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Example 1: Just one source and one area

class { 'apt-sources':
        area    => 'main',
        sources => 'http://backports.deban.org/debian-backports',
}

# Example 2:  Using arrays

class { 'apt-sources':
        area    => ['main','non-free'],
        sources => ['http://backports.deban.org/debian-backports',
          'http://www.deb-multimedia.org']
}

This approach is quick and useful, and does the work. Just a bunch of ideas to improve it: Keyring management, Management of areas in apt-sources.d, and apt-sources as a puppet type.

Recovering a VDI Disk

| Comments

Recently I ran into an issue regarding a vm’s storage. It turns out one of the VDI on my virtual machine was faulty. I had some data inside and I didn’t want to lose it.

First of all, we can convert from VDI to raw. I did the conversion from Windows, I guess it should be the same from Linux.

converting 
1

 C:/VDIs/VboxManage internalcommands converttoraw  disk-vm-testing.vdi  vdisk.raw

Now let’s see what is inside the raw disk:

fdisk 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

dm@testing:#{~} fdisk  -l vdisk.raw
You must set cylinders.
You can do this from the extra functions menu.

Disk vdisk.raw: 0 MB, 0 bytes
255 heads, 63 sectors/track, 0 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000d598a

    Device Boot      Start         End      Blocks   Id  System
vdisk.raw1   *           1         996     7993344   83  Linux
Partition 1 does not end on cylinder boundary.
vdisk.raw2             996        1045      392193    5  Extended
Partition 2 has different physical/logical endings:
     phys=(1023, 254, 63) logical=(1044, 52, 32)
vdisk.raw5             996        1045      392192   82  Linux swap / Solaris

In this case I only have a first partition, because the second one was used as swap device. I have to find out the offset where the data is.

offset 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

dm@testing:#{~}  parted vdisk.raw
GNU Parted 2.3
Using /media/sf_Downloads/vdisk.raw
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) unit
Unit?  [compact]? B
(parted) p
Model:  (file)
Disk /media/sf_Downloads/vdisk.raw: 8589934592B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start        End          Size         Type      File system     Flags
 1      1048576B     8186232831B  8185184256B  primary   ext4            boot
 2      8187280384B  8588886015B  401605632B   extended
 5      8187281408B  8588886015B  401604608B   logical   linux-swap(v1)

(parted)q

The ‘Start’ column shows me the offset for the partition I’m interested in. Next step is to map this offset with a loopback device and mount it.

mount
1
2
3
4
5
6
7

dm@testing:#{~} losetup -o 1048576  /dev/loop0  vdisk.raw
dm@testing:#{~} mount /dev/loop0 /mnt/vdi
dm@testing:#{~} mount | grep loop
/dev/loop0 on /mnt/vdi type ext4 (rw)
dm@testing:#{~} ls /mnt/vdi
ls /mnt/
bin  boot  dev  etc  home  initrd.img  lib  lost+found  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var  vmlinuz

At this point you should be able to access your data, or at least some part of it.

My Git Functions

| Comments

After reading An introduction to bash completion I came up with an idea. I thought it would be really nice to have bash completion for all my git repositories. Furthermore I have a couple of functions that help me out to create new repos on the remote server and gitfy the current folder. For the next two functions I assume public key is enabled.

The function below takes a list of directories and creates bare respositories on the remote server.

g 
1
2
3
4
5
6
7
8
9
10
11
12
13
14

#############################
# Create several repostiories
#############################

g() {
   [ $# -lt 1 ] && printf 'repo name missing\n' && return 126 ;
   for((i=1; i <= $#; i++))
   do
        eval repo='$'$i
        ssh git@server.com  " mkdir ${repo}.git && cd ${repo}.git && git init --bare &> /dev/null"
   done

 echo "${repo}.git" >> $HOME/.repolist
}

In order to keep a list of my repositories, every time I create a new repository I append it to my repolist. Sometimes I have been working on something and I want to add the current directory to a remote repository, here is another function that helps:

gitfy 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

#############################
# Gitfy the current directory
#############################

gitfy() {
   g ${PWD##*/}

cat <<-EOF > .gitignore
*~
*#
EOF

   git init . && git add . &&\
                 git commit -m "First commit" && \
                 git remote add origin ssh://git@server.com/home/git/${PWD##*/}.git && \
                 git push origin master

}

Here is the bash completion for my git repositories :

clone 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

##############################
# Repositories bash completion
##############################
clone() {
        git clone ssh://git@server.com/home/git/${1}
}

_clone() {

    local cur prev opts
    COMPREPLY=()
    cur="${COMP_WORDS[COMP_CWORD]}"
    prev="${COMP_WORDS[COMP_CWORD-1]}"
    opts=$( cat $HOME/.repolist )

    if [[ ${cur} == * ]] ; then
        COMPREPLY=( $(compgen -W "${opts}" ${cur}) )
        return 0
    fi

}
complete -F _clone clone

And the example :

example 
1
2
3
4
5

adm@testing:${~}clone [TAB]
mygits.git             puppet-template.git    tcp.git
capistrano.git         course-saas.git        dotfiles.git
ldap.git               notes.git              puppet-vps.git
todos.git

I’m quite sure it could be much better but for me works fine. It’s pretty handy, and I had the opportunity to play around with bash completion.

Strace to the Rescue

| Comments

Today in the IRC somebody asked which would be the best way to know if a process already exists in the system. The choice was between using test -d /proc/PID or kill -0 PID.

Both of them do the job, the question here, we want to use the best one. Suddenly I remembered an option that comes with strace and lets you query the number of syscalls for a given trace. Besides we can order based on the number of syscalls.

cmd1 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

adm@testing:${~} strace -c -S calls kill -0 1234
kill: No such process
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0        12           mmap2
  -nan    0.000000           0         6           close
  -nan    0.000000           0         5           open
  -nan    0.000000           0         5           fstat64
  -nan    0.000000           0         4           read
  -nan    0.000000           0         4         4 access
  -nan    0.000000           0         3           brk
  -nan    0.000000           0         3           munmap
  -nan    0.000000           0         2           mprotect
  -nan    0.000000           0         1           write
  -nan    0.000000           0         1           execve
  -nan    0.000000           0         1           getpid
  -nan    0.000000           0         1         1 kill
  -nan    0.000000           0         1           dup
  -nan    0.000000           0         1         1 _llseek
  -nan    0.000000           0         1           fcntl64
  -nan    0.000000           0         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                    {52}       6 total

On the other hand, the second command’s output:

cmd2 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

adm@testing:${~}strace -c -S calls test -d /proc/1234
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
  -nan    0.000000           0         7           mmap2
  -nan    0.000000           0         5           close
  -nan    0.000000           0         3           open
  -nan    0.000000           0         3         3 access
  -nan    0.000000           0         3           brk
  -nan    0.000000           0         3           fstat64
  -nan    0.000000           0         2           mprotect
  -nan    0.000000           0         1           read
  -nan    0.000000           0         1           execve
  -nan    0.000000           0         1           munmap
  -nan    0.000000           0         1         1 stat64
  -nan    0.000000           0         1           set_thread_area
------ ----------- ----------- --------- --------- ----------------
100.00    0.000000                   {31}         4 total

The column calls helps to know the number of performed syscalls. Using test -d /proc/PID gives a better performance due to a minor number of syscalls.

I really like strace, is a tool you had better know, here I got syscalls statistics, but you can trace syscalls either specific or a bunch of them, follow forked processes and much more, this is only a simple example. I hope it helps.

RMbackups

| Comments

The first time I came up with this script was a long time ago and I did it with Perl. Today I feel like doing such a thing using Ruby.

Basically the script takes a number of directories and an optional pattern (last argument) from the command line. If either a directory or a pattern is not providen, it looks for in the current directory, and use a default pattern.

The script searches for those files matching the pattern and deletes thoses files from the system. Besides, it checks the owner of the files is the same that is running the script.

rmbackups.rb 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

#!/usr/bin/env ruby
require 'find'
directories = []

pattern = /.*~$/
if ! ARGV.empty?  and ARGV.size > 1
  pattern = /#{ARGV.pop}/
end

directories = (ARGV.empty?) ? File.expand_path(File.dirname(__FILE__)) : ARGV
directories.each do |d|

  Find.find("#{d}") do |file|
    if file.match(pattern)
      if File.owned?("#{file}")
         puts "Deleting #{file}"
   File.delete("#{file}")
      end
    end

  end
end

The default pattern is set to delete backup files ending with ”~”. Below is the ouput:

shell 
1
2
3
4

dev@testing$(~) ./rmbackups.rb
Deleting /home/dev/ruby/findit.rb~
Deleting /home/dev/ruby/delicious.rb~
Deleting /home/dev/ruby/rmbackups.rb~

You might change the pattern from the command line:

shell 
1
2
3
4
5

.dev@testing$(~) /rmbackups.rb /var/tmp /tmp  ".?#.*#$"
Deleting /var/tmp/#.vile.somethin#
Deleting /var/tmp/.#strangefile#
Deleting /tmp/#file#
Deleting /tmp/#a-strangefile#

This is how it works, I did not try too many different patterns beyond these.

Interesting Shell

| Comments

Finally here is my last compilation about shell environment variables.

1.CDPATH

If this is set, keeps a directory list separated by ‘:’ and every time that you type the command “cd DIR” it searches for the directory in the above list, even if your current directory is not the right one. Let’s see an example, I define the directory holding all my git repositories, if I try to make cd repo from any part on the system, even if I not in the right place, I will get there:

shell 
1
2
3
4
5

export CDPATH="~/mygits"
devadm@testing$(~)ls mygits
blog scripts org-mode cv
devadm@testing$(/usr/share/doc) cd  blog
devadm@testing$(~/mygits/blog)

In my opinion is useful to keep just one directory or probably up to two, more than this, it could get messy.

2.FIGNORE

If you want to narrow down the ouput when performing filename completion, this is without a doubt your shell env. When set, it ignores suffixes while performing filename completion.

shell 
1
2
3
4
5

devadm@testing$(~)export FIGIGNORE="#:.o:~"
devadm@testing$(~/application)ls file*
file1.o file2.o file1 file1 file~
devadm@testing$(~)ls [TAB] [TAB]
file1 file2

I mentioned this shell env because it was curious to me. I guess I have never found any case where I had to use such a thing, but that does not mean it wouldn’t be useful to somebody else.

3.HISTCONTROL

It can be set to ignorespace , and it does not record words starting with empty space. On the other hand I do not like to have duplicated lines, ingnoredups does not repeat entries in the history. Finally, I like to join both options so I use ignoreboth instead.

.bashrc
1

export HISTCONTROL=ignoreboth

4.HOSTFILE

It holds the path to a file containing a list of hosts in the same format as /etc/hosts , if it is set, tries to complete the hostname with one of the entries on the file. Otherwiese will look for /etc/hosts.

It could be pretty useful if you want to keep a personal file with your hosts.

.bashrc 
1

export HOSTFILE="~/.myhosts"

The only problem I see here, if I you want to access hosts defined in /etc/hosts. You could make a script that checks if there is a new entry in the /etc/hosts and then appends the last entries to your personal list. Once again, this is only an idea I came up with, but actually I did not follow through.

5.TMPDIR

If set, bash uses its value as the name of a directory in which creates temporary files. With this option I can set my personal temp directory.

.bashrc 
1
2

 export TMPDIR="~/.tmpbash"

6.TMOUT

Sets a timeout and affects to read or select builtin commands, when not input is given. An interesting case is if you set this to your current shell. After N seconds without providing any input it will kill your shell, so keep this in mind.

tmout-read.sh 
1
2
3
4
5
6
7
8
9

#!/bin/bash

# Sets to 3 seconds timeout
TMOUT="3"

printf 'Could you please give me an absolute path ?'
read -r -s -n10 absolute_path

[ -z name ] && printf 'Your path is : %s\n' $absolute_path

Well, the last posts were focused on bash shell environment because I was digging into the man bash and I found quite interesting things I did not know, I hope some of them were useful for you as well.

Shell Environment Variables

| Comments

In the previous post I wrote about different topics such as BASH_ENV, subshells or expressions. Today I’m going to talk about shell enviroment variables.

1. BASHPID

This is the PID of the current bash process. This behaviour is different from $$ in cases such a subshell where $BASHPID says the PID of the subshell, whereas $$ shows the PID of the bash holding the subshell.

bashpid.sh 
1
2

$ echo $$ $BASHPID # 23353 23353
$ echo 'Subsshell' $(echo $$; echo $BASHPID)# 21060 23353

2. BASH_LINENO

Number of lines in the current script, from the beginning to the line the function was called from.

bash_lineno.sh 
1
2
3
4
5
6
7
8
9
10
11

#!/bin/bash 

function show_env_vars()

  echo "$FUNCNAME"    # show_env_vars
  echo "$BASH_LINENO" # Number on lines till show_env_vars was call (11)
  echo "$LINENO"      # Current line 7

}

show_env_vars

3. DIRSTACK

Array with the directories you are moving using popd and pushd builtins to add/remove directories.

dirstack.sh 
1
2
3
4
5
6
7
8
9
10

# After doing some pushd 

$ for i in  ${DIRSTACK[@]}; do printf  'DIR: %s\n' $i; done                                                                                                                                                               

DIR: /tmp
DIR: /var/tmp/testing
DIR: /home/cartoon
DIR: /usr/share/doc
DIR: /var/tmp
DIR: /var/www

4. EUID and GROUPS

EUID of the current user, initialized at shell startup. This variable is read only. On the other hand GROUPS is an array of groups which the current user is member of.

euid_groups.sh 
1
2
3

for ((i=0; i < ${#GROUPS[@]};  i++)); do printf 'gid:%d\n' ${GROUPS[$i]}" ; done
gid: 1002
gid: 1001

5. Gathering OS info

Below there are some of the shell environment vars you could use, for instance if you need to check the architecture. I have not tried in Sparc yet but I would like.

1
2
3
4

HOSTNAME='Self explanatory'
HOSTTYPE='Arhcitecture i486'.
OSTYPE='Operating System where bash is running i.e: linux-gnu'
PPID='Parent process ID of the current shell'

6. Random numbers

1
2
3
4

RANDOM A random number betwwen 0 and 32767.
# Getting a random number between 0 and 99

echo "RANDOM: $(( $RANDOM % 100))

In sum, there are a wide number of shell environment variables to keep in mind if you are scripting. Next time I will post more shell vars, but focused on customizing your bash shell.

Nifty Bash Scripting

| Comments

One of the things I should do more often is to read the man pages. Recently I have spent some time digging into the bash man pages, and I would like to share some interesting stuff.

1.BASH_ENV

If you run a script, it looks for this variable and if it’s set, expands the value to the name of a file and reads its content. The filename should content the absolute pathmane otherwise it will no be able to locate the file.

.bashrc 
1

export BASH_ENV="$HOME/.customs"

Take a look to the above example. From now, and beacuse I set BASH_ENV in my .bashrc, my scripts will have a set of fuctions or whatever I defined in there. It would be the same doing source $HOME/.customs in every script. This is really good, you have a set of custom functions or variables available to all your scripts.

A good idea would be to set this variable in either /etc/profile or /etc/bash.bashrc , if you want to share it with the rest of users in the system.

2.Run in a subshell ( ) vs current shell{ }

This is someting you might not pay attention. In order to assign the ouput of some shell commands to a variable, try first to use { }. Let’s see an example:

colors.sh 
1
2
3
4
5
6
7
8
9

# Current shell
declare -A colors=();

line=''
{ for key in ${!colors[@]}; do line+="$key" ; done; }

# Subshell
line=
$(for key in ${!colors[@]}; do line+="$key" ; done;)

Perhaps you realized the problem it crops up here.

Current shell {}

-Variables are available while the script is running. In the previus example I can access both variables, colors and line. Commands are separated by ‘;’. It’s also quite handy when using conditionals.

1

 if_true  &&  { cmd1; cmd2; cmd3; }

Subshell $()

-Variable assigments do not remain. Any change to any variable in the script would not take effect after ending the command execution. However it would be possible to get the output by means of either echo or printf commands. Besides variable scope, performance could be worse due to new subshell execution.

3.Using [[ ]] expressions

You might know the old form [ ] , but this is the new one, and has some pretty cool properties as Pattern Matching. I guess the best way to get an idea is watching an example.

bash_regex.sh 
1
2
3
4
5
6
7
8
9
10

ip_address="10.20.30.40"
[[ $ip_address =~ ^([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})\.([0-9]{1,3})$ ]] && echo "Valid IP address"

# Iterating matches
for n in ${BASH_REMATCH[@]}; do echo $n; done
10.20.30.40
10
20
30
40

Of course the above example is just for purpose, actually it does not validate a real ip address, but does the trick. Most interesting is the variable BASH_REMATCH, an array holding each substring matched by parenthesized subexpressions. Regarding regular expressions you should look for in the man as in regex(3) and regex(7).

4. Case and ;& operator

This operator continues the execution to next option if available. I came up with an example and you will see how it works:

pickcolor.sh 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

options="ac:lh"
while getopts $options  flag
do
        case $flag in
                a)
                        list_all="on"
                        ;&
                c)

                        c=${OPTARG,,}
                        [ $list_all == "on" ]  && shift
                        [ $list_all == "off" ] && shift && shift

                        args="$*"
                        msg="${args:-$default}"


                        if [ $list_all == "on" ];then
                                for color in ${!colors[@]}
                                do
                                        draw_with_color "$color" "$msg"
                                done
                        else
                                check_color "$c" && draw_with_color "$c" "$msg"
                        fi

                           ;;
                l|-list)
                        printf 'Available colors: \n'
                        _show_colors && exit
                        ;;
                h|-help)
                        usage && exit
                        ;;
                *)
                        usage && exit
                        ;;
        esac
done

Previus chunk of code is part of the pickcolor script. The main idea was to jazz some text up with a chosen color, then I thought it would be more practical to enable the option of pating the same text with all the available colors.

1
2
3
4
5
6
7
8
9
10
11
12
13
14


pickcolor.sh
usage: pickcolor.sh [OPTIONS] message
Options:
       -a --all    Test all colors for message
       -c --color  Set font color.
       -l --list   List available colors.
       -h --help   Help

# Colors up using all the colors
$ pickcolor -a  Show me the colors

# Color up using just one color
$ pickcolor -c red Paint my room !!

The special operator ;& was really useful and did the trick. Next part I will write about shell vars and some builtin commands to have in mind.