Probe Server
Server that probes endpoints over HTTP & HTTPS and AWS S3 buckets, producing metrics in JSON or for Prometheus.
Inspired by Prometheus Blackbox Exporter.
- Usage
- Metrics
- HTTP probe
- S3 probe
- Versioning policy
Usage
Visit http://probe-srv.herokuapp.com/?target=http://google.com&pretty=true for a sample probe of an HTTP endpoint.
The target
query parameter indicates what to probe:
- An HTTP(S) endpoint, e.g.
https://google.com
, to use the HTTP probe - An S3 bucket URL, e.g.
s3://bucket_name
, to use the S3 probe
The response will be a JSON object with:
- A
success
boolean property indicating whether the probe was successful. - A
metrics
array of objects, describing various metrics collected from the probed target, such as duration of the probe, HTTP status code, number of S3 objects, etc. - A
failures
array which may indicate possible causes of the probe's failure.
Installation
Run it with Docker
docker run -p 3000:3000 alphahydrae/probe-srv
Or, run it with npx
npx probe-srv
Or, install and run it manually
npm install -g probe-srvprobe-srv
Then, try it
Visit http://localhost:3000?target=http://google.com&pretty=true
Metrics
The server's probes provide various metrics about their target, mostly numeric values such as counts, bytesizes, last modification dates, etc.
Each metric is a JSON object with the following format:
The following metric types exist at this time:
- boolean -
true
orfalse
. - bytes - Integer greater than or equal to zero.
- datetime - Date in ISO-8601 format.
- number - Number with no associated unit of measurement.
- quantity - Integer greater than or equal to zero, representing an amount of something.
- seconds - Number greater than or equal to zero.
The value of a metric may be null
if it cannot be determined (e.g. the
httpCertificateExpiry
metric can be null
if the
HTTP request is not made over TLS, or the s3LastObjectModificationDate
metric can be null
if the S3 bucket contains
no objects).
Prometheus format
To get the metrics in Prometheus's text format, use the /metrics
path:
http://probe-srv.herokuapp.com/metrics?target=http://google.com
Metric names are converted from the JSON's camel-case format to underscored
format and the probe_
prefix is prepended (e.g. httpStatusCode
becomes probe_http_status_code
). Additionally, the metric's type is added as
a suffix if it's bytes
or seconds
(e.g. duration
becomes
probe_duration_seconds
).
Metric tags are added as labels.
All metrics will be provided as gauges by applying the following conversions by type:
- boolean -
true
becomes1
andfalse
becomes0
. - bytes - No conversion, except for
null
which becomes-1
. - datetime - Dates are provided in Unix
time, i.e. the number of seconds
that have elapsed since 00:00:00 UTC on Thursday, 1 January 1970.
null
becomes-1
. - number - No conversion, except for
null
which becomesNaN
. - quantity - No conversion, except for
null
which becomes-1
. - seconds - No conversion, except for
null
which becomes-1
.
Generic metrics
The following sub-headings document metrics that are provided by all probes.
duration
Type: seconds
How long the probe took to complete in seconds.
HTTP probe
The HTTP probe is used when the target is an URL that starts with http://
or
https://
. By default, it will make a GET
request to that URL, following any
redirects, and provide various metrics about the HTTP response.
HTTP probe metrics
The following sub-headings document the metrics provided by the HTTP probe.
httpCertificateExpiry
Type: datetime
Expiration date of the SSL certificate (when the target starts with https://
).
httpContentLength
Type: bytes
Length of the HTTP response in bytes.
httpDuration
Type: seconds, Repeated
Duration of the HTTP request(s) by phase, summed over all redirects, in seconds. Several instances of this metric will be provided, one for each request phase:
dnsLookup
- How long it took to perform the DNS lookup.tcpConnection
- How long it took to open the TCP connection after the DNS lookup.tlsHandshake
- How long it took to perform the TLS handshake for an HTTPS probe, after the TCP connection was established. (This metric will be absent for URLs starting withhttp://
.)firstByte
- How long it took to receive the first byte of the response after the TLS handshake (or TCP connection).contentTransfer
- How long it took to transfer the entire response entity once the first byte was received.
httpRedirects
Type: quantity
Number of times HTTP 301 or 302 redirects were followed.
httpSecure
Type: boolean
Indicates whether SSL/TLS was used for the final redirect.
httpStatusCode
Type: number
HTTP status code of the final response.
httpVersion
Type: number
HTTP version of the final response.
HTTP probe parameters
The following sub-headings document URL query parameters that can be provided to customize the behavior of the HTTP probe.
allowUnauthorized
Type: boolean, Default: false
Whether to consider an HTTP response with an invalid SSL certificate as a success.
?allowUnauthorized=true
followRedirects
Type: boolean, Default: true
Whether the probe will follow redirects (e.g. HTTP 301 Moved Permanently or HTTP 302 Found) to provide metrics about the final response, or whether it will simply provide metrics about the first response sent by the server.
?followRedirects=false
method
Type: string (GET
, POST
, PUT
, etc.), Default: GET
The HTTP method to use for the request on the target URL. GET
by default.
?method=POST
header
Type: key=value
pair, Repeatable
HTTP header to add to the probe's request(s). This parameter can be repeated to set multiple headers.
// The value is "Authorization=Basic YWRtaW46Y2hhbmdlbWUh", URL-encoded
?header=Authorization%3DBasic%20YWRtaW46Y2hhbmdlbWUh
HTTP probe expectations
The following sub-headings document URL query parameters that can be provided to customize how the HTTP probe will determine if it was successful.
By default, it only expects that the final HTTP response will have a code in the 2xx or 3xx range.
expectHttpRedirects
Type: boolean or integer
For the probe to be considered successful with this parameter:
- If
true
, at least 1 redirect must have been followed. - If
false
, no redirect must have been followed. - If an integer, exactly that number of redirects must have been followed.
?expectHttpRedirects=true
?expectHttpRedirects=2
Possible failures:
expectHttpRedirectTo
Type: URL
For the probe to be considered successful with this parameter:
- At least 1 redirect must have been followed.
- The protocol, host, port and path of the final HTTP request's URL must correspond to the specified URL. (Other parts of the URL, such as authentication, query string or hash, are ignored for the comparison.)
// The value is "http://example.com/path", URL-encoded
?expectHttpRedirectTo=http%3A%2F%2Fexample.com%2Fpath
Possible failures:
expectHttpResponseBodyMatch
Type: regular expression, Repeatable
For the probe to be considered successful with this parameter, the HTTP response body must match the regular expression.
This parameter can be repeated to check the presence of multiple patterns.
// The value is "Catch \d{2}", URL-encoded
?expectHttpResponseBodyMatch=Catch%20%5Cd%7B2%7D
Possible failures:
expectHttpResponseBodyMismatch
Type: regular expression, Repeatable
For the probe to be considered successful with this parameter, the HTTP response body must not match the regular expression.
This parameter can be repeated to check the absence of multiple patterns.
// The value is "connection lost", URL-encoded
?expectHttpResponseBodyMismatch=connection+lost
Possible failures:
expectHttpSecure
Type: boolean
For the probe to be considered successful with this parameter:
- If
true
, the final HTTP response must have been over TLS (e.g.https://
). - If
false
, the final HTTP response must not have been over TLS (e.g.http://
).
By default, either is considered successful.
Note that this does not affect the probe's behavior of failing if an SSL
certificate is invalid. Use the allowUnauthorized
parameter for that.
?expectHttpSecure=true
Possible failures:
expectHttpStatusCode
Type: number or HTTP status code class (e.g. 2xx
), Repeatable, Default: [ "2xx", "3xx" ]
For the probe to be considered successful with this parameter, the final HTTP
response's status code must be one of the expected codes, or fall within one of
the expected classes (e.g. 204
falls within the 2xx
class). Both individual
codes and code classes may be provided.
?expectHttpStatusCode=204
?expectHttpStatusCode=200&expectHttpStatusCode=3xx
Possible failures:
expectHttpVersion
Type: number
For the probe to be considered successful with this parameter, the HTTP version of the final response must match the expected version.
?expectHttpVersion=1.1
Possible failures:
HTTP probe failures
The following sub-headings document the possible causes of failure that may be included in the HTTP probe's result.
httpResponseBodyMismatch
The body of the final HTTP response did not match some of the regular
expressions provided with the expectHttpResponseBodyMatch
parameter.
This failure will be repeated for each regular expression that did not match.
insecureHttp
The final HTTP request was expected to be over TLS due to the expectHttpSecure
parameter being set to true
, but it was not.
invalidHttpRedirectCount
An integer was provided as the expected number of redirects with the
expectHttpRedirects
parameter, and the actual number
of followed redirects until the final HTTP response did not match that
expectation.
invalidHttpRedirectLocation
The final redirection was expected to be made to the URL specified with the
expectHttpRedirectTo
parameter, but a different URL
was provided by the server.
invalidHttpStatusCode
The HTTP status code of the final response did not match any of the expected
codes or code classes provided with the expectHttpStatusCode
parameter (or set by default).
invalidHttpVersion
The HTTP version of the final response was not the expected version provided
with the expectHttpVersion
parameter.
missingHttpRedirect
The expectHttpRedirects
parameter was set to true
,
but no redirect was issued by the server.
unexpectedHttpRedirect
The expectHttpRedirects
parameter was set to false
,
but the server issued one or more redirects.
unexpectedHttpResponseBodyMatch
The body of the final HTTP response matched some of the regular expressions
provided with the expectHttpResponseBodyMismatch
parameter.
This failure will be repeated for each regular expression that matched.
The failure object's actual
property will contain the full regular expression
match.
unexpectedlySecureHttp
The final HTTP request was expected not to be over TLS due to the
expectHttpSecure
parameter being set to false
, but it
was.
S3 probe
The S3 probe is used when the target is an URL that starts with s3://
. The
format is as follows:
s3://[access_key_id:secret_access_key@]bucket_name[/prefix]
For example:
s3://my_bucket
s3://A28sdf8A:oa83ufozsr8b@secure_backups/daily
By default, the probe will list all the objects in the bucket and provide
various metrics about these objects' sizes and last modification dates. It can
also be configured to list object versions with the s3Versions
parameter.
S3 probe metrics
The following sub-headings document the metrics provided by the S3 probe.
s3FirstObjectModificationDate
Type: datetime
The modification date of the earliest modified object.
s3FirstObjectVersionModificationDate
Type: datetime
The modification date of the earliest modified object version.
Note: this metric will only be provided if the [s3Versions
parameter] is
set to true
.
s3LargestObjectSize
Type: bytes
The size of the largest object in bytes.
s3LastObjectModificationDate
Type: datetime
The modification date of the most recently modified object.
s3LastObjectVersionModificationDate
Type: datetime
The modification date of the most recently modified object version.
Note: this metric will only be provided if the [s3Versions
parameter] is
set to true
.
s3ObjectsCount
Type: quantity
Number of objects.
s3ObjectsTotalSize
Type: bytes
Total size of objects.
s3ObjectVersionsCount
Type: quantity
Number of object versions.
Note: this metric will only be provided if the [s3Versions
parameter] is
set to true
.
s3SmallestObjectSize
Type: bytes
The size of the smallest object in bytes.
S3 probe parameters
The following sub-headings document URL query parameters that can be provided to customize the behavior of the S3 probe.
s3ByPrefix
Type: string, Repeatable
When provided N
times, this parameter will cause each metric to be provided N + 1
times in the response:
- Once for each prefix (
N
) - Once for no prefix (
+ 1
)
?s3ByPrefix=daily&s3ByPrefix=monthly
For example, if the parameter is provided twice with the values daily
and
monthly
like in the example above, the s3ObjectsTotalSize
metric will be
present twice (as will all other metrics of the S3 probe). Those separate metric
objects can be differentiated by their tags
property:
,,
s3Versions
Type: boolean, Default: false
Whether to list objects versions in the bucket and generate metrics for them.
?s3Versions=true
Versioning policy
This project follows the rules of semantic versioning v2.0.0. Check the CHANGELOG for breaking changes between major versions.
Beware: changes to the description
property of metrics or failures will
not be considered breaking.
Please file a bug If you notice an actual breaking change without a corresponding major version number change.