Description
Link-Metadata used to get metadata from internet URLs (htmls, images, media..). It also returns security policy information (Cross-Origin Resource Sharing, Content Security Policy) and checks them with Google Safe Browsing (Web Risk).
Installation
$ npm install @bedrock-api/link-metadataUsage
Link-Metadata is TypeScript-friendly and comes with both CommonJS and ESM module support.
const lmd = require('@bedrock-api/link-metadata');or
import lmd from '@bedrock-api/link-metadata';First, create a new user if you don't have one:
const firstName = 'name';
const lastName = 'surname';
const email = 'email@some.com';
const username = 'username';
const password = 'password';
const reply = await lmd.createUser(firstName, lastName,
email, username, password);
console.log(reply);Second, log in and create a new access key. If you already have one you don't need to login, just use the key.
const response = await lmd.login(username, password);
if (response.error) {
console.error(response.error);
return;
}
const key = response.accessKey;Getting a metadata for the given URL:
const metadata = await lmd.getMetadata(url, key);API
login(username, password)
Used to get first instance of access key. Returns KeyResponse:
interface AccessKey {
accessId: string;
accessSecret: string;
}
interface KeyResponse {
username?: string;
accessKey?: AccessKey;
error?: string;
}If an error occurs, it is returned in the corresponding field.
createKey(key)
Issue a new key (using current). All keys are the same in terms of functionality, there is no 'primary' key. You may create as much keys as you want. Returns KeyResponse.
getKeys(key)
List all available keys. Returns KeysResponse:
interface KeysResponse {
username?: string;
accessKeys?: AccessKey[];
error?: string;
}revokeKey(key)
Revokes a key (the one that passed in argument). Useful when a key becomes compromised. Returns a string.
getMetadata(url, key)
To obtain metadata for the URL. Needs an access key (token) to be provided. Returns MetadataResponse structure.
interface MetadataResponse {
url: string; // The original request url
status: number | null; // Request status code. Null in cases where url is invlaid and the like. Never make the network request
type: string | null; // The type of the content. This should be a static list (enum). This may be base on
// the file such as video. Or more specific like article/book/etc. Full list to be determined.
contentType: string | null; // HTML mime time from the http request
contentEncoding: string | null; // HTML content encoding from the http request
audio: Audio | null; // If the file is an audio file, audio details will be here.
file: File | null; // If the file is not any of the other supported types, what details we have will be here.
html: HTML | null; // If the file is an html file, html details will be here.
image: Image | null; // If the file is an image file, image details will be here.
video: Video | null; // If the file is an video file, video details will be here.
xml: XML | null; // If the file is an xml file, xml details will be here.
// errors
error: number | null; // Number indicates there was an error and is the error code
reason: string | null; // Reason string for error
warnings: string[] | null; // A list of warnings/errors while processing. See [Warnings](#warnings) section for some examples
// Web Risk
googleSafeBrowsing: GoogleSafeBrowsing | null;
}reportBadData(url, reason)
Reports issues with the metadata (not sufficient, irrelevant, etc). The report should consist of the url of issue and an optional reason message. This helps us to get feedback and improve our service.
const response = await lmd.reportBadData('https://www.youtube.com/watch?v=2mNwZHcoW_o', 'wrong language');What it does
Our service helps you extract metadata without manually processing it on your end. You can detect malware, get all data in one place and store frequently used websites in cache.
SDK installation
Create an empty project in your IDE. Run the following in terminal:
npm install @bedrock-api/link-metadataRun the following (don't forget to insert your credentials):
import { AccessKey, MetadataResponse, login, createKey, getKeys, revokeKey, getMetadata } from '@bedrock-api/link-metadata';
const username = 'User';
const password = 'password';
async function main() {
let key: AccessKey;
let key2: AccessKey;
let meta: MetadataResponse;
// Logging in and creating a new AccessKey
const response = await login(username, password);
if (response.error) return;
key = response.accessKey;
// Getting a metadata for the given URL
const url = 'https://www.npmjs.com';
meta = await getMetadata(url, key);
console.log(meta);
// Creating additional key (optional)
const reply = await createKey(key);
if (reply.error) return;
key2 = reply.accessKey;
// Getting a list of current keys
const keys = await getKeys(key);
if (keys.error) return;
console.log(keys);
// Deleting created keys
await revokeKey(key);
await revokeKey(key2);
}
main();Usage
Log in and try to get a sample response by running this:
const lmd = require('@bedrock-api/link-metadata');
const username = 'User';
const password = 'password';
async function main() {
// Logging in and creating a new AccessKey
let response = await lmd.login(username, password);
const key = response.accessKey;
// Getting a metadata for the given URL
const url = 'https://www.npmjs.com';
response = await lmd.getMetadata(url, key);
console.log(response);
// Creating additional key
response = await lmd.createKey(key);
const key2 = response.accessKey;
// Getting a list of current keys
response = await lmd.getKeys(key);
console.log(response);
// Deleting both created keys
await lmd.revokeKey(key);
await lmd.revokeKey(key2);
}
main();Why use an SDK?
Using SDK allows you to get metadata without the need to encrypt your credentials.
Using the service without SDK
In order to use the service without an SDK installation, you have to use https://www.base64encode.org/ to generate the encoded key. Provide your secret to the service in this format:
accessId:accessSecretThen proceed in Postman by using the key you get from Base64 in Headers section of Authorization.
When you're authorized, the request can be made by adding a URL after url= parameter:
http://localhost:3000/SpinItOn?url=https://shorturl.at/AT089
Sample response
Let's say your name is John Doe and you're maintaining a startup that collects all news connected to the election system. You're running the link to the news article through our service and this would be the response that you get:
{
"url": "https://shorturl.at/AT089",
"status": 200,
"type": "html",
"contentType": "text/html; charset=utf-8",
"contentEncoding": null,
"audio": null,
"file": null,
"html": {
"url": "https://www.foxnews.com/politics/pains-pump-could-shift-midterm-outcome-election-won-lost-pump",
"size": 135327,
"title": "Pain at the pump could shift midterm outcome: 'This election is likely won or lost at the pump' | Fox News",
"description": "Gas prices are back on the rise just weeks before the midterm elections, a development that could decide which party takes control of the House and the Senate this fall.",
"author": "Aubrie Spady",
"publisher": "Fox News",
"language": null,
"languageAlternates": [],
"websiteName": "Fox News",
"image": {
"url": "https://a57.foxnews.com/static.foxnews.com/foxnews.com/content/uploads/2022/03/1200/675/gas-pumps-3.jpg?ve=1&tl=1",
"size": 127729,
"format": "jpg",
"height": 675,
"width": 1200,
"security": {
"cors": {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Methods": "GET,HEAD",
"Access-Control-Allow-Headers": "*",
"Access-Control-Max-Age": "86400"
},
"csp": {},
"x-frame-options": null
}
},
"video": null,
"audio": null,
"favicon": {
"url": "https://static.foxnews.com/static/orion/styles/img/fox-news/favicons/favicon.ico",
"size": 10990,
"format": "ico",
"height": 48,
"width": 48,
"security": {
"cors": {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Methods": "GET,HEAD,POST",
"Access-Control-Allow-Headers": "*",
"Access-Control-Max-Age": "86400"
},
"csp": {},
"x-frame-options": null
}
},
"security": {
"cors": {
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Methods": "GET,HEAD",
"Access-Control-Allow-Headers": "*",
"Access-Control-Max-Age": "86400"
},
"csp": {},
"x-frame-options": "DENY"
}
},
"image": null,
"video": null,
"xml": null,
"error": null,
"reason": null,
"warnings": null,
"googleSafeBrowsing": {
"isUrlSafe": true,
"result": {}
}
}Fields content
You should expect the following fields that contain metadata. If there’s no content - that might happen due to lack of such data on the original website.
-
URL name- the link you’ve run to get metadata -
Status- request result code (standardized: ex. 404 error) -
Type- type of content we’ve got following your link -
Image- shows if the link contains any images and provides the link to it -
ContentType- the type of content that is used in this link -
ContentEncoding- ? -
Audio- shows if the link contains any audio files and provides the link to it -
File- shows if the link contains any files and provides the link to it -
HTML- shows the metadata on the requested HTML page -
Size- shows the size of the content -
Format- shows the format of the content -
Height- shows height of the image -
Width- shows width of the image -
Video- shows if the link contains an video files and provides the link to it -
Xml- shows if the link contains an XML file and provides the link to it -
Error- critical errors that interfere with running the request -
Reason- the explanation of the error -
Warnings- non-critical yet important notes regarding your request -
GoogleSafeBrowsing- shows if the website you’re running has potential dangers such as malware, viruses e.t.c.