Showing posts with label Bash. Show all posts
Showing posts with label Bash. Show all posts

Friday, 10 November 2017

[Bash / Google search] How to query google search in bash ?

Recently my colleague asked me if I could write some script that automatically asks google about some games. He has a list of game titles in a file like:
➜  tools cat products 
Witcher 3
Metro Last Light
Valiant Hearts

And now he wants to ask google where those games can be bought:
where can I buy Witcher 3
where can I buy Metro Last Light
where can I buy Valiant Hearts

and then print first 5 links of each result.
At first I was thinking about writing scala script but I was also wondering if I can do the same in bash.
I've found an app called googler. You can find it on github: https://github.com/jarun/googler
It allows to use google search engine from the command line. You can use either interactive mode or json. For instance I want first two results for "scala cookbook":
googler "scala cookbook" --count 2 --json

It returns:
[
  {
    "abstract": "Save time and trouble when using Scala to build object-oriented, functional, and 
concurrent applications. With more than 250 ready-to-use recipes and 700 code ...",
    "title": "Scala Cookbook - O'Reilly Media",
    "url": "http://shop.oreilly.com/product/0636920026914.do"
  },
  {
    "abstract": "Save time and trouble when using Scala to build object-oriented, functional, and 
concurrent applications. With more than 250 ready-to-use recipes and 700 code ...",
    "title": "Scala Cookbook: Recipes for Object-Oriented and Functional ...",
    "url": "https://www.amazon.com/Scala-Cookbook-Object-Oriented-Functional-Programming/dp/1449339611"
  }
]

And this is exactly what I needed. I want only links so I used jq in order to extract url field from json objects like this:
➜  tools googler "scala cookbook" --count 2 --json | jq '.[].url'
"http://shop.oreilly.com/product/0636920026914.do"
"https://www.amazon.com/Scala-Cookbook-Object-Oriented-Functional-Programming/dp/1449339611"

Note that I used .[].url because it returns array of objects.
Now I want to loop over game titles from the file, print the title and results from google:
➜  tools cat products | while read title; do echo $title && googler "where can I buy $title" --count 2 --json | jq '.[].url'; done
Witcher 3
"http://buy.thewitcher.com/"
"https://www.gog.com/game/the_witcher_3_wild_hunt"
Metro Last Light
"http://store.steampowered.com/app/287390/Metro_Last_Light_Redux/"
"https://www.g2a.com/metro-last-light-redux-steam-key-global-i10000000633010?___store=polish"
Valiant Hearts
"https://store.ubi.com/us/valiant-hearts---the-great-war/575ffdb2a3be1633568b4e7a.html"
"https://www.microsoft.com/pl-pl/store/p/valiant-hearts-the-great-war/c0x15tr4np0b"

And the last thing - allow user to pass the query (where can I buy) and number of expected links and make it executable:
#!/bin/bash

QUERY=$1
RESULTS=$2

cat products | while read title; do echo $title && googler "$1 $title" --count $2 --json | jq '.[].url'; done

It works exactly as I expected so there's really no need to use any more sophisticated programming language. I realize that I could do the same using curl but googler has a lot of useful features so you should definitely try it.

Friday, 23 September 2016

[Bash / Git] How to print number of commits done today ?

Quite simple stuff. You want to know how many commits you've done so far today. How to do that ?
➜  cw git:(develop) git log
It shows commits in the following format:
commit 2affec1442eb6b4ad21cd6d93c7670e49e4f3cba
Author: gt
Date:   Wed Sep 21 13:33:15 2016 +0200

    got rid of some warnings

commit 2bd8a52b239301adedeee8a36cff47c2599f992f
Author: gt
Date:   Wed Sep 21 13:25:24 2016 +0200

    bumped scala version in versions.gradle
You can exactly see when and by whom the commit has been made.
Author: gt
Date:   Wed Sep 21 13:25:24 2016 +0200
We need to filter git log output by author and date. It would be easier to grep the data if both author and commit were in the same line. We also need date only (time isn't important here) so let's format git log output like that:
git log --date=short --format=format:"%ad %aE %s"
It prints:
2016-09-23 mymaila@mycompany.pl commit message 1
2016-09-23 mymaila@mycompany.pl commit message 2
2016-09-23 mymaila@mycompany.pl commit message 3
2016-09-23 mymaila@mycompany.pl commit message 4
2016-09-23 mymaila@mycompany.pl commit message 5
2016-09-23 mymaila@mycompany.pl commit message 6
2016-09-23 mymaila@mycompany.pl commit message 7
2016-09-22 mymaila@mycompany.pl commit message 8
2016-09-22 mymaila@mycompany.pl commit message 9
2016-09-22 mymaila@mycompany.pl commit message 10
Now we can grep such output easily but we still need to know current date in the same format.
➜  cw git:(develop) date +'%Y-%m-%d'
2016-09-23
Let's put it together:
➜  cw git:(develop) ✗ git log --date=short --format=format:"%ad %aE %s" | grep "$(date +'%Y-%m-%d') mymaila@mycompany.pl"
2016-09-23 mymaila@mycompany.pl commit message 1
2016-09-23 mymaila@mycompany.pl commit message 2
2016-09-23 mymaila@mycompany.pl commit message 3
2016-09-23 mymaila@mycompany.pl commit message 4
2016-09-23 mymaila@mycompany.pl commit message 5
2016-09-23 mymaila@mycompany.pl commit message 6
2016-09-23 mymaila@mycompany.pl commit message 7
And the final step - count the lines:
➜  cw git:(develop) ✗ git log --date=short --format=format:"%ad %aE %s" | grep "$(date +'%Y-%m-%d') mymaila@mycompany.pl" | wc -l
8
I guess it's faster to count manually commits in git log than type the whole command above so let's put it into a script:
#!/bin/bash

TODAY=$(date +'%Y-%m-%d')
AUTHOR="mymaila@mycompany.pl"
COMMITS=$(git log --date=short --format=format:"%ad %aE %s" | grep "$TODAY $AUTHOR" | wc -l)

echo "Commits today: $COMMITS"
I've also added alias in .zshrc:
alias hmc="/home/gt/tools/howManyCommitsToday.sh"
➜  cw git:(develop) hmc
Commits today: 8

Monday, 8 February 2016

]Bash / Gawk] How to print how long do you have to stay at work today ?

Today I'm going to show you a short script wirtten when I was running integration tests. It simply shows minutes you still have to be at work.
#!/bin/bash
START_HOUR=$1
START_MINUTE=$2

if [ -z "$START_HOUR" ] || [ -z "$START_MINUTE" ]; then
    START_HOUR="9"
    START_MINUTE="45"
fi

date | gawk '{print $4}' | gawk -F":" '{print 8 * 60 - (($1 * 60 + $2) - ('"$START_HOUR"' * 60 + '"$START_MINUTE"'))}'
It's actually very simple.
START_HOUR=$1
START_MINUTE=$2
Here I create two variables which contain hour and minute I came to the office.
[ -z "$START_HOUR" ]
This condition checks whether a variable contains some value. In case I don't pass the date when I came it sets my default which is 9:45. Then very simple oneliner:
date | gawk '{print $4}' | gawk -F":" '{print 8 * 60 - (($1 * 60 + $2) - ('"$START_HOUR"' * 60 + '"$START_MINUTE"'))}'
date
prints current date -> Mon 8 Feb 15:12:09 CET 2016
date | gawk '{print $4}'
prints the fourth field (split by whitespace) -> 15:12:55
gawk -F":"
-F allows you to specify how do you want the input string to be split - in this case it's ':' so $1 now contains current hour and $2 current minute.
'"$START_HOUR"'
this is how you can access shell variables in gawk And then some simple math (note that I assume that working day == 8h):
'{print 8 * 60 - (($1 * 60 + $2) - ('"$START_HOUR"' * 60 + '"$START_MINUTE"'))}'
8 * 60 = working day (minutes)
($1 * 60 + $2) - ('"$START_HOUR"' * 60 + '"$START_MINUTE"'))
current minute of day minus minute I came to the office
The result of the script is: 148 which means I can go home after 148 minutes :)
You can obviously pass what time you came to work:
./howLong 10 0 prints 162

Sunday, 17 May 2015

[Bash / ssh] How to invoke command remotely without password / private key prompts ?

Sometimes you may want to invoke command on remote machine via ssh. You can obviously pass a command in double quotes:
ssh root@somehost.com "echo \$HOME"
This example prints to the console a value of env variable HOME (note that the dollar sign has to be escaped otherwise HOME variable will be resolved on your local machine). Let's say I want to fetch a value of some env variable in my bash script which will be started by Jenkins. There are actually two problems:
  • ssh will prompt for password,
  • if you haven't already accepted host's key there will be another prompt.
If you don't have proper entry in ~/.ssh/known_hosts you will see:
gt ~ ssh root@somehost.com "echo \$HOME"
                                                        
The authenticity of host 'somehost.com (10.92.30.38)' can't be established.
RSA key fingerprint is b0:c6:ad:6b:06:73:a3:de:31:8c:f8:4d:07:4e:2c:e6.
Are you sure you want to continue connecting (yes/no)? 

so you need to type "yes" in order to invoke the command.
In case you've already accepted the key you will see only:
gt ~ ssh root@somehost.com "echo \$HOME"                                                                                                           
root@somehost.com's password: 
Ssh doesn't have any flag for password (security) so you cannot do something like:
ssh root@somehost.com -p mySecretPassword
Solution for that is sshpass. I'm sure it's available in your linux distribution's repository. On Fedora install it using:
sudo yum install sshpass
So now you can pass the password easily:
gt ~ sshpass -p mySecretPassword ssh root@somehost.com "echo \$HOME"                                    
/root
In case you need to accept host's key you can use ssh -oStrictHostKeyChecking=no. Example:
gt ~ ssh root@somehost.com "echo \$HOME"                                                                          
The authenticity of host 'somehost.com (10.92.30.38)' can't be established.
RSA key fingerprint is b0:c6:ad:6b:06:73:a3:de:31:8c:f8:4d:07:4e:2c:e6.
Are you sure you want to continue connecting (yes/no)? ^C
zsh: interrupt  ssh root@somehost.com "echo \$HOME"
gt ~ sshpass -p mySecretPassword ssh -oStrictHostKeyChecking=no root@somehost.com "echo \$HOME"                             
Warning: Permanently added 'somehost.com,10.92.30.38' (RSA) to the list of known hosts.
/root
There is sctually another way of importing keys - ssh-keyscan command which output has to be appended to ~/.ssh/known_hosts file.
ssh-keyscan -H somehost.com >> ~/.ssh/known_hosts
Example:
gt ~ ssh root@somehost.com          
The authenticity of host 'somehost.com (10.92.30.39)' can't be established.
RSA key fingerprint is 3d:7d:a0:82:d7:3b:60:bc:58:ce:14:d2:bf:1e:d5:89.
Are you sure you want to continue connecting (yes/no)? ^C
zsh: interrupt  ssh root@somehost.com
gt ~ ssh-keyscan -H somehost.com >> ~/.ssh/known_hosts  
# somehost.com SSH-2.0-OpenSSH_5.3
# somehost.com SSH-2.0-OpenSSH_5.3
no hostkey alg
gt ~ ssh root@somehost.com 
Warning: Permanently added the RSA host key for IP address '10.92.30.39' to the list of known hosts.
root@somehost.com's password: 
Last login: Thu May 14 15:24:06 2015 from 10.154.8.71
[root@somehost ~]#
Value returned by the command invoked on remote host can obviously be assigned to some variable in bash script:
gt ~ cat script.sh
#!/bin/bash
REMOTE_HOME=$(sshpass -p arthur ssh -oStrictHostKeyChecking=no root@somehost.com "echo \$HOME")
echo "remote home = ${REMOTE_HOME}"
gt ~ ./script.sh
remote home = /root
As you can see both problems can be solved quite easily but you should realize that this kind of hacks (sshpass) shouldn't be used in production environment. Actually I use this kind of scripts which pass password in plain text only in test environments which aren't directly connected to the internet. Generally such machines are used only for snapshots' testing and don't store any crucial data. You should definitely read this part of sshpass man page:
SECURITY CONSIDERATIONS

First and foremost, users of sshpass should realize that ssh's insistance on only getting the password interactively is not without reason. 
It is close to impossible to securely store the password, and users of sshpass should consider whether ssh's public key authentication provides the same end-user experience, while involving less hassle and being more secure.

The -p option should be considered the least secure of all of sshpass's options. 
All system users can see the password in the command line with a simple "ps" command. Sshpass makes a minimal attempt to hide the password, but such attempts are doomed to create race conditions without actually solving the problem. 
Users of sshpass are encouraged to use one of the other password passing techniques, which are all more secure.

In particular, people writing programs that are meant to communicate the password programatically are encouraged 
to use an anonymous pipe and pass the pipe's reading end to sshpass using the -d option. 

Monday, 17 September 2012

[bash / find] How to list 10 largest jar files ?

 Sometimes you may need to list largest or smallest files of given type. Find seems to be a perfect tool to perform such tasks. Let's say I want to find 10 largest .jar files in my local maven repository. The repository contains 2498 jars.

gt ~/.m2/repository find . -name "*.jar" | wc -l                                                                                                      [1497] 
2498
This task can be completed in 4 simple steps:

1. Find all jar files:
gt ~/.m2/repository find . -type f -name "*.jar"                                                                                          [1499] 
./commons-pool/commons-pool/1.6/commons-pool-1.6.jar
./commons-pool/commons-pool/1.5.7/commons-pool-1.5.7.jar
./xalan/xalan/2.7.1/xalan-2.7.1.jar
./xalan/serializer/2.7.1/serializer-2.7.1.jar
./joda-time/joda-time/2.3/joda-time-2.3.jar
./joda-time/joda-time/2.1/joda-time-2.1.jar
./xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar
...
-type f - means that find looks only for flat files
-name "*.jar" - looks for files which end with .jar

2. Print out the size of each file:
gt ~/.m2/repository find . -type f -name "*.jar" -exec du -h {} \;                                                                                    [1502] 
112K ./commons-pool/commons-pool/1.6/commons-pool-1.6.jar
100K ./commons-pool/commons-pool/1.5.7/commons-pool-1.5.7.jar
3,1M ./xalan/xalan/2.7.1/xalan-2.7.1.jar
272K ./xalan/serializer/2.7.1/serializer-2.7.1.jar
568K ./joda-time/joda-time/2.3/joda-time-2.3.jar
560K ./joda-time/joda-time/2.1/joda-time-2.1.jar
1,2M ./xerces/xercesImpl/2.9.1/xercesImpl-2.9.1.jar
1,4M ./xerces/xercesImpl/2.11.0/xercesImpl-2.11.0.jar
...
-exec du -h {} - executes du -h on each result ({} - result placeholder)

3. Sort files by size:
gt ~/.m2/repository find . -type f -name "*.jar" -exec du -h {} \; | sort -hr                                                                         [1506] 
46M ./org/glassfish/extras/glassfish-embedded-all/3.0.1/glassfish-embedded-all-3.0.1.jar
25M ./com/vaadin/vaadin-client-compiler-deps/1.0.2/vaadin-client-compiler-deps-1.0.2.jar
22M ./com/censored
21M ./com/liferay/portal/portal-impl/6.1.0/portal-impl-6.1.0.jar
20M ./org/robotframework/robotframework/2.8.3/robotframework-2.8.3.jar
16M ./com/vaadin/vaadin-client/7.1.0/vaadin-client-7.1.0.jar
...
In this step all the results returned by find are being piped to sort.
-r - reversed order
-h - human readable form

4. Show only 10 files:
gt ~/.m2/repository find . -type f -name "*.jar" -exec du -h {} \; | sort -hr | head -n 10                                                            [1507] 
46M ./org/glassfish/extras/glassfish-embedded-all/3.0.1/glassfish-embedded-all-3.0.1.jar
25M ./com/vaadin/vaadin-client-compiler-deps/1.0.2/vaadin-client-compiler-deps-1.0.2.jar
22M ./com/censored
21M ./com/liferay/portal/portal-impl/6.1.0/portal-impl-6.1.0.jar
20M ./org/robotframework/robotframework/2.8.3/robotframework-2.8.3.jar
16M ./com/vaadin/vaadin-client/7.1.0/vaadin-client-7.1.0.jar
16M ./com/censored
14M ./org/scala-lang/scala-compiler/2.11.5/scala-compiler-2.11.5.jar
14M ./org/scala-lang/scala-compiler/2.10.3/scala-compiler-2.10.3.jar
13M ./com/cenqua/clover/clover/3.1.2/clover-3.1.2.jar
I'm pretty sure that there's linux command which does the same but on the other hand this short example shows how powerful find is.