Friday 10 November 2017

[Bash / Google search] How to query google search in bash ?

Recently my colleague asked me if I could write some script that automatically asks google about some games. He has a list of game titles in a file like:
➜  tools cat products 
Witcher 3
Metro Last Light
Valiant Hearts

And now he wants to ask google where those games can be bought:
where can I buy Witcher 3
where can I buy Metro Last Light
where can I buy Valiant Hearts

and then print first 5 links of each result.
At first I was thinking about writing scala script but I was also wondering if I can do the same in bash.
I've found an app called googler. You can find it on github: https://github.com/jarun/googler
It allows to use google search engine from the command line. You can use either interactive mode or json. For instance I want first two results for "scala cookbook":
googler "scala cookbook" --count 2 --json

It returns:
[
  {
    "abstract": "Save time and trouble when using Scala to build object-oriented, functional, and 
concurrent applications. With more than 250 ready-to-use recipes and 700 code ...",
    "title": "Scala Cookbook - O'Reilly Media",
    "url": "http://shop.oreilly.com/product/0636920026914.do"
  },
  {
    "abstract": "Save time and trouble when using Scala to build object-oriented, functional, and 
concurrent applications. With more than 250 ready-to-use recipes and 700 code ...",
    "title": "Scala Cookbook: Recipes for Object-Oriented and Functional ...",
    "url": "https://www.amazon.com/Scala-Cookbook-Object-Oriented-Functional-Programming/dp/1449339611"
  }
]

And this is exactly what I needed. I want only links so I used jq in order to extract url field from json objects like this:
➜  tools googler "scala cookbook" --count 2 --json | jq '.[].url'
"http://shop.oreilly.com/product/0636920026914.do"
"https://www.amazon.com/Scala-Cookbook-Object-Oriented-Functional-Programming/dp/1449339611"

Note that I used .[].url because it returns array of objects.
Now I want to loop over game titles from the file, print the title and results from google:
➜  tools cat products | while read title; do echo $title && googler "where can I buy $title" --count 2 --json | jq '.[].url'; done
Witcher 3
"http://buy.thewitcher.com/"
"https://www.gog.com/game/the_witcher_3_wild_hunt"
Metro Last Light
"http://store.steampowered.com/app/287390/Metro_Last_Light_Redux/"
"https://www.g2a.com/metro-last-light-redux-steam-key-global-i10000000633010?___store=polish"
Valiant Hearts
"https://store.ubi.com/us/valiant-hearts---the-great-war/575ffdb2a3be1633568b4e7a.html"
"https://www.microsoft.com/pl-pl/store/p/valiant-hearts-the-great-war/c0x15tr4np0b"

And the last thing - allow user to pass the query (where can I buy) and number of expected links and make it executable:
#!/bin/bash

QUERY=$1
RESULTS=$2

cat products | while read title; do echo $title && googler "$1 $title" --count $2 --json | jq '.[].url'; done

It works exactly as I expected so there's really no need to use any more sophisticated programming language. I realize that I could do the same using curl but googler has a lot of useful features so you should definitely try it.