r7insight_docker
Forward all your logs to the Rapid7 Insight Platform, like a breeze.
Arguments
Mandatory arguments
Token
You can supply the tokens using command line arguments:
-
--logstoken
: Log token for logs -
--statstoken
: Log token for stats -
--eventstoken
: Log token for events -
--token
: Log token which is used for the above tokens if one is not provided.- You can supply only this token and it'll be used for logs, stats and events.
You can also supply log, stats and event tokens using environment variables.
When both command line arguments and environment variables are supplied, the command line arguments are used.
-
INSIGHT_LOGSTOKEN
: Log token for logs -
INSIGHT_STATSTOKEN
: Log token for stats -
INSIGHT_EVENTSTOKEN
: Log token for events -
INSIGHT_TOKEN
: Log token which is used for any of the above tokens if one is not provided.- You can only supply this token and it'll be used for logs, stats and events.
Region
You need to supply the region to forward your logs to with either of these arguments:
-r <REGION>
--region <REGION>
E.g. -r 'eu'
Optional arguments
-
--no-stats
if you do not want stats to be published to the Rapid7 Insight Platform every second. You need this flag for Docker version < 1.5 -
--no-logs
if you do not want logs to be published to the Rapid7 Insight Platform. -
--no-docker-events
if you do not want events to be published to the Rapid7 Insight Platform. -
-i, --statsinterval <STATSINTERVAL>
downsamples the logs sent to the Rapid7 Insight Platform. It collects samples and averages them before sending. -
-a/--add
allows you to add a fixed value to the data being published. This follows the format<name>=<value>
.- If you don't use the
-a
flag, a default value ofhost="$(uname -n)"
will be added. - You cannot supply multiple
-a
flags
- If you don't use the
-
--no-secure
if you want your logs to be sent to the Insight Platform un-encrypted (no TSL/SSL). -
--log-level
to specify the logging level for the r7insight_docker container itself.- Can also be enabled by specifying the environment variable
INSIGHT_LOG_LEVEL
E.g.INSIGHT_LOG_LEVEL=info
- By default the logger is silent unless a logger level is defined
- Can also be enabled by specifying the environment variable
Filtering
You can also filter the containers for which logs/stats are forwarded.
Notes
- Please ensure you correctly escape the regex pattern since these can be expanded/interpreted by your shell; ideally in single-quotes.
- Use
'.*nginx.*'
rather than.*nginx.*
- Use
- Do not supply the REGEX pattern in a LEQL fashion as in the UI, filtering is based on normal regular expressions.
- Use
'.*nginx.*'
, not'/.*nginx.*/'
, note the removed foreslashes.
- Use
- For each of the arguments below you can only supply at most one of each.
-
Use
--skipByName '.*(nginx|haproxy).*'
-
Do not use
--skipByName '.*nginx.*' --skipByName '.*haproxy.*'
-
Filters
-
--matchByName '<REGEX>'
: forward logs/stats only for the containers whose name matches the given<REGEX>
. -
--matchByImage '<REGEX>'
: forward logs/stats only for the containers whose image matches the given<REGEX>
. -
--skipByName '<REGEX>'
: do not forward logs/stats for the containers whose name matches the given<REGEX>
. -
--skipByImage '<REGEX>'
: do not forward logs/stats for the containers whose image matches the given<REGEX>
.
Usage as a Container
The simplest way to forward all your container logs to the Rapid7 Insight Platform is to run this repository as a container.
- If you are using this container in production please ensure that you pin the version of the image.
- Use
rapid7/r7insight_docker:1.2.3
rather thanrapid7/r7insight_docker:latest
sincelatest
might pull in breaking changes if a new version is released - You can see the the available versions here.
- Use
Docker Arguments
The --read-only
docker flag specifies that the container file system will be read-only.
This is not a requirement but since currently there's no need for writing, it makes the container more secure.
The --security-opt=no-new-privileges
docker flag sets a kernel bit which stops the process or its children
from gaining additional privileges via setuid or sgid.
Once again not required, but increases security.
Examples
docker run -v /var/run/docker.sock:/var/run/docker.sock \
--read-only \
--security-opt=no-new-privileges \
rapid7/r7insight_docker \
-t <TOKEN> \
-r <REGION> \
-j \
-a host="$(uname -n)"
You can also use different tokens for logging, stats and events:
docker run -v /var/run/docker.sock:/var/run/docker.sock \
--read-only \
--security-opt=no-new-privileges \
rapid7/r7insight_docker \
-l <LOGSTOKEN> \
-k <STATSTOKEN> \
-e <EVENTSTOKEN> \
-r <REGION> \
-j \
-a host="$(uname -n)"
Running in a restricted environment
Some environments (such as Google Compute Engine) do not allow to access the Docker socket without special privileges.
You will get EACCES(Error: read EACCES
) error if you try to run the container.
To run the container in such environments add --privileged
to the docker run
command.
Example:
docker run --privileged \
-v /var/run/docker.sock:/var/run/docker.sock \
--read-only \
--security-opt=no-new-privileges \
rapid7/r7insight_docker \
-t <TOKEN> \
-r <REGION> \
-j \
-a host="$(uname -n)"
Usage as a CLI
$ npm install r7insight_docker -g
$ r7insight_docker -t "$TOKEN" -r "$REGION" -a host="$(uname -n)"
Embedded usage
Install it with: npm install r7insight_docker --save
Then, in your JS file:
const insight = require('r7insight_docker');
const logger = insight.utils.start({
json: false, // or true to parse lines as JSON
secure: true, // or false to connect over plain TCP
region: "eu", // specify region
token: process.env.TOKEN, // Insight Platform TOKEN
newline: true, // Split on newline delimited entries
stats: true, // disable stats if false
add: null, // an object whose properties will be added
// the following options limit the containers being matched
// so we can avoid catching logs for unwanted containers
matchByName: /hello/, // optional
matchByImage: /matteocollina/, //optional
skipByName: /.*pasteur.*/, //optional
skipByImage: /.*dockerfile.*/ //optional
});
// logger is the source stream with all the
// log lines
setTimeout(function() {
logger.destroy();
}, 5000);
Building
Docker
We currently release the container on two different bases:
- Node on Debian Bullseye
- Node on Alpine Linux
Using the Docker file
First clone this repository, then:
# For Debian Bullseye base:
docker build -t r7insight_docker .
# Or for Alpine Linux base:
docker build -t r7insight_docker -f alpine.Dockerfile .
Make
Firstly
# For Debian Bullseye base, also the default if not specified:
export BUILD_TYPE=node-bullseye
# Or for Alpine Linux base:
export BUILD_TYPE=node-alpine
Then:
make build
make test
If you've build and tested and want to push:
# Default is "rapid7/r7insight_docker"
export DOCKER_REGISTRY_PREFIX=<your-dockerhub-user>/<your-image-name>
make tag
make push
Node
- Update package.json depending on your requirements
make publish
How it works
This module wraps four Docker APIs:
-
POST /containers/{id}/attach
, to fetch the logs -
GET /containers/{id}/stats
, to fetch the stats of the container -
GET /containers/json
, to detect the containers that are running when this module starts -
GET /events
, to detect new containers that will start after the module has started
This module wraps docker-loghose and docker-stats to fetch the logs and the stats as a never ending stream of data.
All the originating requests are wrapped in a never-ending-stream.