Find files and tar them

/note to myself

Finding files with most powerful utility find sending them to tarball

  find . -maxdepth 2 -type f -print0 | tar -czvf backup.tar.gz --null -T -  

It will:

• deal with files with spaces, newlines, leading dashes, and other funniness
• handle an unlimited number of files
• won't repeatedly overwrite your backup.tar.gz like using tar -c with xargs will do when you have a large number of files

source here

Parallel SVN up for workspace with svn:externals

Yo, svn users!
Say we have a workspace with lots of svn projects. And workspace is related to the project via svn:externals property.


Here is a nice short script that saves a lot of time performing svn update in parallel:

#!/bin/bash

set -x

if [ -z "$1" ]

then

SVN_REV=""

else

SVN_REV="-r $1 "

fi

svn up --ignore-externals ${SVN_REV} .

svn pg svn:externals . | \

  grep -v "^$" | \

  sed -e "s/\^//g" | \

  awk '{ print "svn co https://my_repo.com"$1" "$2 }' | \

  sed -e 's/$/ \&/g' \

  > _svn_up_parallel

cat _svn_up_parallel

bash _svn_up_parallel

rm _svn_up_parallel

script description:

  grep -v "^$"  - removes empty strings from svn:externals

  sed -e "s/\^//g"  - removes ^ from lines

  awk '{ print "svn co https://my_repo.com"$1" "$2 }'  - converts record from 

           "/my_project1/trunk my_project1" to "https://my_repo.com/my_project1/trunk my_project1"

  sed -e 's/$/ \&/g' - inserts a " &" at the end of each line to send the command to background

Automating tasks over telnet.

Ok. So it is a Stone Age and no one knows what SSH is. Everybody is using telnet.

Let's automate a service restarting task.

Cool and easy as 1,2,3.

{ \

 sleep 2; \

 echo "root"; \

 sleep 2; \

 echo "password"; \

 sleep 2; \

 echo "ps ax | grep my_cool_service"; \

 sleep 2; \

 echo "OLD_SERVICE_PID=\`ps ax | grep my_cool_service | awk '{print \$1}'\` && kill \$OLD_SERVICE_PID"; \

 sleep 2; \

 echo "service my_cool_service start && ps ax |grep my_cool_service"; \

 sleep 4; \

 } | telnet 10.10.10.10

  

Capturing Screen shots on Android via ADB

Let's use screencap tool on Android to capture a screen shot and save it to a file on Android's internal /sdcard.

adb shell screencap -p /sdcard/myScreenShot.png

Ok, lets capture a series of screen shots - a movie.

adb shell mkdir -p /sdcard/moovie1; 

for i in `seq -w 1 1 155`; do \

  adb shell screencap -p /sdcard/moovie1/cap_$i.png && \

  echo -e "cap_$i.png"; \

done;

Now let's download the images:

for i in `seq -w 1 1 155`; do \

  adb pull /sdcard/moovie1/cap_$i.png && \

  adb shell rm /sdcard/moovie1/cap_$i.png && \

  echo -e "cap_$i.png\n"; \

done;

Ok, now let's make a movie with help of well-known imagemagick tool::
Scale images to 50% size and convert them to GIF format.

for i in `ls`; \
  do convert $i -scale 50% $i.gif; \
done;

convert -delay 100 -loop 0 cap*.gif animated_screen.gif

Also one could use following script to save captured images directly to computer

 adb shell screencap -p | sed 's/\r$//' > screen.png 

or 

adb shell screencap -p | perl -pe 's/\x0D\x0A/\x0A/g' > screen.png

to cat images directly to a file on a computer (src)

(but for series of screen shots that will be a bit slower).

adb shell screencap -h

usage: screencap [-hp] [-d display-id] [FILENAME]

   -h: this message

   -p: save the file as a png.

   -d: specify the display id to capture, default 0.

If FILENAME ends with .png it will be saved as a png.

If FILENAME is not given, the results will be printed to stdout.

A strange way to populate SSH keys from Jenkins to Jenkins Slave node

Using sshpass tool as a workaround for ssh interactive password prompt. :-)

mkdir tmpz

cd tmpz

wget --output-document=sshpass.tgz "http://downloads.sourceforge.net/project/sshpass/sshpass/1.05/sshpass-1.05.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fsshpass%2Ffiles%2Fsshpass%2F&ts=1380045506&use_mirror=freefr"

tar -zxvf sshpass.tgz

cd sshpass-1.05

./configure

make

./sshpass -p "hudson"  ssh-copy-id -i ~/.ssh/id_rsa.pub hudson@10.116.65.165

Use perl to edit multiple lines XML

Ok, here is a strange request - to edit some xml file over ssh on Mac OS X.

One needs to change false to true in following

<key>system.privilege.taskport</key> <dict> <key>allow-root</key> <false/>

Luckily, Mac OS X has perl (not sure if it was installed along with XCode). So let's use perl for that:

ssh user@192.168.1.2 "set -x; \
echo 'Password' | sudo -S \
perl -p -0777 -i -e 's@<key>system.privilege.taskport</key>.*?<dict>.*?<key>allow-root</key>.*?<false/>@<key>system.privilege.taskport</key>\n\t\t<dict>\n\t\t\t<key>allow-root</key>\n\t\t\t<true/>@sg' /etc/authorization"

Breaking this down:

• -p tells it to loop through the input and print
• -0777 tells it to use the end of file as the input separator, so that it gets the whole thing in in one slurp
• -e means here comes the stuff I want you to do

And the substitution itself:

• use @ as a delimiter so you don't have to escape /
• use *?, the non-greedy version, to match as little as possible, so we don't go all the way to the last occurrence of </xyz> in the file
• use the s modifier to let . match newlines (to get the multiple-line tag values)
• use the g modifier to match the pattern multiple times

Perl expression was constructed with help of this answer from stackoverflow.

Using tar pipe for copying directory excluding files or directories.

A nice use case for tar pipe

Say we have a directory, with tons of subdirectories, each of which is a checked subversion project.

Dir structure looks like this.

RootDir / 

  /dir1/

    .svn

    dirs....

  /dir2/

    .svn

    dirs....

One wants to copy everything but .svn dirs.

Let's use tar pipe for that: 

tar --exclude \.svn -c RootDir | tar -C /path/to/destination/ -xv

Start process, get PID of just started process, and terminate the process by PID later

Ok, let's say we have a usecase - a job on Jenkins CI that needs to start a background process, remember PID of the process, pass the PID to some later cleanup. And we need something that will work on Linux and on Mac OS X (yes, even this OS can behave polite).

So 1st, let's start a process and remember it's PID. This is done via $! shell environment variable. The $! variable will have PID of latest started process. So here is an example:

nohup tail -f /var/log/syslog > /tmp/$BUILD_TAG.log & export LOGGER_PID=$! 
# $BUILD_TAG - is an environment variable provided by Jenkins CI, it has unique job execution identifier

2nd, let's remember the PID

echo -e "LOGGER_PID=$LOGGER_PID" > /tmp/$BUILD_TAG

3rd, let's use the PID and send SIGTERM to the process if it is still alive. We'll wrap it to a shell script.

#!/bin/bash
. /tmp/$BUILD_TAG  # read to LOGGER_PID env variable

kill_proc(){
  local pid_alive=`ps ax| awk '{print $1}' | grep -e "^$LOGGER_PID$" >>/dev/null; echo $?`;
   # ps ax| awk '{print $1}' - will print to stdout PIDs without surrounding spaces
   # grep -e "^$LOGGER_PID$" >>/dev/null; - will search for the PID, ^ and $ are important, just in case our PID is a substring of some other PID on system
   # echo $? - will echo exit code of last command
  if [ $pid_alive -eq 0 ]; then 
    kill $LOGGER_PID; # sending  SIGTERM to the process
    sleep 10;
  else
    echo "Proc with PID $LOGGER_PID already terminated" 
  fi;  
}

kill_proc;