webche
TypeScript icon, indicating that this package has built-in type declarations

0.1.2 • Public • Published

Overview

Webche provides a node library to scrape basic details of a url using the metascraper and node-unfluff libraries.

Getting Started

Install

Install

npm i webche --save or yarn add webche

Usage

import { scrape } from 'webche';
// or
const { scrape } = require('webche');
// define a url
const url = "http://motherboard.vice.com/read/google-wins-trial-against-oracle-saves-9-billion"
const { metascraper, unfluff } = await scrape(url, lazy=true)
 
console.log(metascraper)
 
{
    audio: null,
    author: 'Sarah Jeong',
    logo:
    'https://images.vice.com/motherboard/content-images/article/no-id/1464294050922700.jpg',
    publisher: 'VICE',
    date: '2016-05-26T20:14:00.000Z',
    description:
    'The ruling that Google’s use of APIs is a fair use will have wide-ranging ramifications for the rest of the tech industry.',
    image:
    'https://images.vice.com/motherboard/content-images/article/no-id/1464294050922700.jpg?crop=0.847111111111111xw:1xh;center,center&resize=1200:*',
    lang: 'en',
    title: 'Google Wins Trial Against Oracle, Saves $9 Billion',
    url:
    'https://www.vice.com/en_us/article/kb77gv/google-wins-trial-against-oracle-saves-9-billion',
    video: null
}
 
// The text extraction algorithm can be somewhat slow on large documents. If you only need access to elements like title or image, you can use the lazy extractor to get them more quickly without running the full processing pipeline by setting lazy=true, default value is true. This returns an object just like the regular extractor except all fields are replaced by functions and evaluation is only done when you call those functions.
 
// with lazy = true
console.log(unfluff.title())
 
// with lazy = false
console.log(unfluff.title)
 
console.log(unfluff)
// lazy=false
{
    title: 'Google Wins Trial Against Oracle, Saves $9 Billion',
    softTitle: 'Google Wins Trial Against Oracle, Saves $9 Billion',
    date: '2016-05-26T20:14:00Z',
    author: [ 'Sarah Jeong' ],
    publisher: 'Vice',
    copyright: '2017 VICE Media LLC"',
    favicon:
    'https://vice-web-statics-cdn.vice.com/favicons/vice/favicon.ico',
    description:
    'The ruling that Google\'s use of APIs is a fair use will have wide-ranging ramifications for the rest of the tech industry.',
    keywords:
    'culture, news, lgbtq, politics, journalism, video, documentary, sex, drugs, film, tv, entertainment, travel, crime,tech,Motherboard,copyright,Oracle,motherboard show,fair use,APIs,oracle v. google',
    lang: 'en',
    canonicalLink:
    'https://www.vice.com/en_us/article/kb77gv/google-wins-trial-against-oracle-saves-9-billion',
    tags:
    [ 'Horoscopes',
    'tech',
    'Motherboard',
    'copyright',
    'Oracle',
    'motherboard show',
    'fair use',
    'APIs',
    'oracle v. google' ],
    image:
    'https://images.vice.com/motherboard/content-images/article/no-id/1464294050922700.jpg?crop=0.847111111111111xw:1xh;center,center&resize=1200:*',
    videos: [],
    links: [ [Object], [Object], [Object], [Object] ],
    text:
    'Google just won in Oracle v. Google, a $9 billion case over Android code. At 1:00 PM PDT, a jury of ten people delivered a verdict in favor of Google.\n\nThe lawsuit was first filed in 2010. There was already a trial in 2012, but after an appeal to the Federal Circuit, the parties underwent a second trial over copyrighted code.\n\nAt the end of their third day of deliberation, the jury found that Google\'s use of the declaring code and the structure, sequence, and organization of the Java APIs in the Android code was a fair use.\n\nAfter the verdict was read aloud, Judge William Alsup thanked the jury for their service, noting that the jurors—who often came to court even earlier than the set start time of 7:45 AM, and lingered after hours to pore over their notes—had been "attentive" and "worked hard."\n\n"I salute you for your extreme hard work in this case," he told the jury, which on Tuesday, he had called "the best jury this courthouse has ever seen."\n\nOnce the jury was dismissed, Alsup said, "I know there will be appeals and the like." Oracle is expected to appeal the decision, meaning that this already six-year-long litigation will drag out even longer.\n\nBut still, lawyers for Google were wreathed in smiles after their big victory, laughing and hugging each other as Oracle lawyers huddled grimly on the other side of the courtroom.\n\nThe tech industry has had its eyes on this case, since Google\'s alleged infringement—a clean room reimplementation of the APIs—is a widespread industry practice. Many commentators—and if the tortured analogies that came up at trial are any indication, the lawyers themselves—feared that the jury would not understand the technical issues at the heart of the case. We don\'t know if the jury understood APIs, but the verdict is in: Google\'s use is a fair use.'
}
 
// lazy=true
{
    title: [Function: title],
    softTitle: [Function: softTitle],
    date: [Function: date],
    copyright: [Function: copyright],
    author: [Function: author],
    publisher: [Function: publisher],
    favicon: [Function: favicon],
    description: [Function: description],
    keywords: [Function: keywords],
    lang: [Function: lang],
    canonicalLink: [Function: canonicalLink],
    tags: [Function: tags],
    image: [Function: image],
    videos: [Function: videos],
    text: [Function: text],
    links: [Function: links]
}
 

Extracted data elements

This is what unfluff will try to grab from a web page:

  • title - The document's title (from the <title> tag)
  • softTitle - A version of title with less truncation
  • date - The document's publication date
  • copyright - The document's copyright line, if present
  • author - The document's author
  • publisher - The document's publisher (website name)
  • text - The main text of the document with all the junk thrown away
  • image - The main image for the document (what's used by facebook, etc.)
  • videos - An array of videos that were embedded in the article. Each video has src, width and height.
  • tags- Any tags or keywords that could be found by checking <rel> tags or by looking at href urls.
  • canonicalLink - The canonical url of the document, if given.
  • lang - The language of the document, either detected or supplied by you.
  • description - The description of the document, from <meta> tags
  • favicon - The url of the document's favicon.
  • links - An array of links embedded within the article text. (text and href for each)

This is what metascraper will try to grab from a web page:

Todo

Contributors

Rakesh Paul - Xtrios

License

This project is licensed under the MIT License.

Package Sidebar

Install

npm i webche

Weekly Downloads

0

Version

0.1.2

License

ISC

Unpacked Size

47.6 kB

Total Files

10

Last publish

Collaborators

  • rakeshpaul