wgrep

wgrep is a command line utility for running selector queries against HTTP-based resources.

Examples

wgrep -q "a" http://google.com

This example will return a tab-delimited list of link text and resources they are linked to.

If you want only the a tags that have an href:

wgrep -q "a[href]" http://google.com

The q flag accepts a css selector for text/html (and XML/XHTML) MIME'd resources.

But what if you've got a JSON url?

wgrep -miq "$.feed.entry[*].title[*].$t" "https://gdata.youtube.com/feeds/api/videos?q=cats&v=2&alt=json"

The above will show you an ordered list of the title of the top 25 search results on YouTube for "cats".

We use JSONPath to handle JSON query operations.

Usage

wgrep is your standard command line utility. As such, you run it basically with:

wgrep [options] <url>

An HTTP URL is required (although we are definitely thinking about adding file and stdin inputs as well).

Install

Install using npm - we recommend a global install:

npm install -g wgrep
...
which wgrep
> /usr/local/bin/wgrep

Now you should have a working wgrep command.

Options

• -h, --help: Prints out the help message.

• -V, --version: Prints out the version number.

• -u, --user <username>: Allows you to provide a username for basic auth.

• -p, --password [password]: Allows you to provide a password for basic auth. Will ask for one if it's not provided.

• -q, --query <query>: Allows you to provide a query selector string to run against the resource.

• -m, --markdown: Returns results in a (best-guess) Markdown format.

  This works really nicely with links and images:

  wgrep -mq "a[href]" http://google.com
  wgrep -mq "img" http://google.com

• -i, --ordered: With -m, returns results as an ordered list.

• -l, --list: With -m, returns results as an unordered list.

• -j, --json: Returns results as a JSON string.

• -n, --inspect: With -j, inspects the JSON results, with a default depth of 2 and colorized output.

Credits

Written by Paul DeLeeuw for debugging URLs and other HTTP-based shenanigans.

Written in Javascript, runs in node. Makes use of several awesome open source projects, like

• cheerio
• commander
• JSONPath
• request
• underscore

Thanks to all the contributors of these fine projects.