Advanced

This section covers a variety of advanced topics.

Isolating application components

If you are interested in components from the application layers of your image, but not interested in components from the underlying platform layers, you can exclude components from platform layers from your results.

For example, if you build your application on ubuntu:latest (your Dockerfile starts with FROM ubuntu:latest), you can exclude components from the Ubuntu layer(s) so that the components generated by Black Duck Docker Inspector contain only components from your application layers.

Limitations to this feature

This feature works when the method you use to build your image adds layers on top of a base or platform image (single-stage builds). Here's an example of a Dockerfile that adds layers on top of a base image. This feature will work for images built this way:

FROM ubuntu:latest
RUN apt-get update && apt-get -y install curl

The resulting image will include all layers from ubuntu:latest, plus an additional layer that adds the curl package. This feature is designed for this type of image, and will be able to isolate the layers added to ubuntu:latest (which in this case is a single layer that adds curl).

On the other hand, when you build an image using multiple FROM statements in the Dockerfile (multi-stage builds), the layers in the resulting image are not, in general, the layers of the first image followed by the layers of the second image. You can verify this by comparing the RootFS.Layers list of the resulting image to the RootFS.Layers lists of the images named in the FROM statements. (See below for instructions on how to get the RootFS.Layers list using the docker inspect command).

Here's an example of a Dockerfile that uses multiple FROM statements. This feature will not, in general, work for images built this way:

FROM ubuntu:latest
FROM myapplicationimage:mytag

How to use this feature

First, find the layer ID of the platform's top layer using the following process.

  1. Run the docker inspect command on the platform image. In this example, the platform image is ubuntu:latest, so you would run docker inspect ubuntu:latest.
  2. Find the last element in the RootFS.Layers array. This is the platform top layer ID. In the following example, this is sha256:b079b3fa8d1b4b30a71a6e81763ed3da1327abaf0680ed3ed9f00ad1d5de5e7c. (Because the latest tag moves frequently, the top layer ID for ubuntu:latest changes over time.)

Set the value of the Black Duck Docker Inspector property docker.platform.top.layer.id to the platform top layer ID. For example:

./blackduck-docker-inspector.sh ... --docker.platform.top.layer.id=sha256:b079b3fa8d1b4b30a71a6e81763ed3da1327abaf0680ed3ed9f00ad1d5de5e7c

In this mode, the container file system and/or container file system squashed image produced by Black Duck Docker Inspector only contains files added to the image by application layers. If the Black Duck signature scanner is run on this file, it generates results based only on files found on application layers. This provides the benefit of isolating application components by excluding platform components. However, there may be some loss in match accuracy from the signature scanner because in this scenario, the signature scanner may be deprived of some contextual information, such as the operating system files that enable it to determine the Linux distribution.
This could negatively affect its ability to accurately identify components.

Inspecting multiple images more efficiently by leaving services running

By default, Black Duck Docker Inspector starts, uses, and then stops and removes either one or two containerized image inspector services per run. This may be appropriate when scanning a single image, but when scanning many images it is highly inefficient, and it doesn't support concurrent execution of multiple Black Duck Docker Inspector runs.

The scanning of many images is completed significantly faster by starting the image inspector services once, and running multiple instances of Black Duck Docker Inspector, so that each one sends requests to the already-running image inspector services.

The following script illustrates how this is done in a Docker environment:

curl -O https://raw.githubusercontent.com/blackducksoftware/blackduck-docker-inspector/master/deployment/docker/batchedImageInspection.sh

To keep the example simple, this script only starts the Alpine image inspector service. In general, you must start two more services: the Ubuntu image inspector service for inspecting images built from dpkg-based Linux distros, and the CentOS image inspector service for inspecting images built from rpm-based Linux distributions. It doesn't matter which service receives the request; any service redirects if necessary.

Concurrent execution

You can inspect multiple images in parallel on the same computer if you (a) directly invoke the .jar file, and (b) leave the services running. For example:

# Get the latest blackduck-docker-inspector.sh
curl -O  https://blackducksoftware.github.io/blackduck-docker-inspector/blackduck-docker-inspector.sh
chmod +x ./blackduck-docker-inspector.sh

# Determine the current Black Duck Docker Inspector version
inspectorVersion=$(grep "^version=" blackduck-docker-inspector.sh|cut -d'"' -f2)

# Download the latest Black Duck Docker Inspector .jar file to the current dir
./blackduck-docker-inspector.sh --pulljar

# Execute multiple inspections in parallel
java -jar ./blackduck-docker-inspector-${inspectorVersion}.jar --blackduck.url={Black Duck url} --blackduck.username={Black Duck username} --docker.image=alpine:3.5 --cleanup.inspector.container=false &
# Allow time for the image inspector service to start before starting the rest of the runs (do this for each image inspector type that you use: alpine, centos, ubuntu)
java -jar ./blackduck-docker-inspector-${inspectorVersion}.jar --blackduck.url={Black Duck url} --blackduck.username={Black Duck username} --docker.image=alpine:3.4 --cleanup.inspector.container=false &
java -jar ./blackduck-docker-inspector-${inspectorVersion}.jar --blackduck.url={Black Duck url} --blackduck.username={Black Duck username} --docker.image=alpine:3.3 --cleanup.inspector.container=false &

Alternative methods for setting property values

Black Duck Docker Inspector gets its property values from Spring Boot's configuration mechanism. Black Duck Docker Inspector users can leverage Spring Boot capabilities beyond command line arguments and environment variables; for example, hierarchy of property files and placeholders to manage properties in more sophisticated ways.

Passing passwords and other values with increased security

For greater security, sensitive property values such as passwords can be set through the environment variables using one of the Spring Boot configuration mechanisms previously mentioned.

For example, instead of passing --blackduck.password=mypassword on the command line, you can do the following:

export BLACKDUCK_PASSWORD=mypassword
./blackduck-docker-inspector.sh --blackduck.url=http://blackduck.mydomain.com:8080/ --blackduck.username=myusername --docker.image=ubuntu:latest

Refer to Spring Boot's configuration mechanism for more information.

Air Gap mode

In Black Duck Docker Inspector versions 6.2.0 and higher, Black Duck provides an archive containing all files required to run Black Duck Docker Inspector without requiring access to the internet. To download the Air Gap archive, run the command:

./blackduck-docker-inspector.sh --pullairgapzip

To create the Docker images from the Air Gap archive required by Black Duck Docker Inspector, run the commands:

unzip blackduck-docker-inspector-{version}-air-gap.zip
docker load -i blackduck-imageinspector-alpine.tar
docker load -i blackduck-imageinspector-centos.tar
docker load -i blackduck-imageinspector-ubuntu.tar

To run in Air Gap mode, use the command:

./blackduck-docker-inspector.sh --upload.bdio=false --jar.path=./blackduck-docker-inspector-{version}.jar --docker.tar={tarfile}

Configuring Black Duck Docker Inspector for your Docker Engine and registry

If you invoke Black Duck Docker Inspector with an image reference; in other words, a repo:tag value versus a .tar file, it uses the docker-java library (https://github.com/docker-java/docker-java) to access the Docker registry to pull the image.

If docker pull works from the command line, then Black Duck Docker Inspector is able to pull that image, because docker-java can be configured the same way as the Docker command line utility (docker).

There are also other ways to configure docker-java. For more information on configuring docker-java (and therefore Black Duck Docker Inspector) for your Docker registry, refer to: https://github.com/docker-java/docker-java#Configuration.

Black Duck Docker Inspector does not override any of the configuration settings in the code, so all other methods such as properties, system properties, and system environment, are available to you.

If you use environment variables, and you are calling Black Duck Docker Inspector from Detect, you must prefix the environment variable names with DETECT_DOCKER_PASSTHROUGH_ to instruct Detect to pass them on to Black Duck Docker Inspector. In this scenario, instead of export SOMENAME=value, use export DETECT_DOCKER_PASSTHROUGH_SOMENAME=value.

To use system properties which are normally set using java -D, and if you are calling Black Duck Docker Inspector from Detect, you must put the properties in a file; for example, mydockerproperties.properties, and use:

--detect.docker.passthrough.system.properties.path=mydockerproperties.properties

to point Black Duck Docker Inspector to those property settings.

Running Black Duck Docker Inspector on Open Container Initiative (OCI) images

When given a Docker image, (--docker.image=repo:tag), Black Duck Docker Inspector uses the docker-java library equivalent of docker save to save the image to a tar file. In this scenario, Black Duck Docker Inspector can pull, save, and inspect any image that can be pulled using a docker pull command. Since Black Duck Docker Inspector uses the docker-java library, the Docker client executable does not actually need to be installed.

When given a saved Docker tarfile, (--docker.tar=image.tar), Black Duck Docker Inspector requires a Docker Image Specification v1.2.0 format file. To inspect Open Container Initiative (OCI) format image files, Synopsys recommends using skopeo to convert them to Docker Image Specification v1.2.0 files. For example, the command:

skopeo copy oci:alpine-oci docker-archive:alpine-docker.tar

converts an OCI image directory alpine-oci to a Docker Image Specification v1.2.0 format file alpine-docker.tar that Black Duck Docker Inspector can process when passed in with the --docker.tar=alpine-docker.tar command line argument.

Running Detect on a project directory that exists within a Docker image

When you want to run Detect on a directory that exists within a Docker image, you can use the following approach: 1. Run Detect on the image to generate the container filesystem for the image. 2. Run Detect on a directory within that container filesystem.

Detect performs these actions without running the image/container.

To see a simple example that illustrates this approach, use the following commands to download these sample files:

curl -O https://raw.githubusercontent.com/blackducksoftware/blackduck-docker-inspector/master/deployment/docker/runDetectInImageDir/runDetectInImageDir.sh
curl -O https://raw.githubusercontent.com/blackducksoftware/blackduck-docker-inspector/master/deployment/docker/runDetectInImageDir/Dockerfile

Review the script before running it to make sure the side effects (files and directories that it creates) are acceptable. You must make the script executable before you run it.

Running the signature scanner on a specific directory within a Docker image

To use iScan to scan a specific directory within an image:

  1. Run Black Duck Docker Inspector on the target image to get the container file system. You can also do this using Detect using --detect.docker.passthrough.* properties. Include the following Black Duck Docker Inspector properties:
--upload.bdio=false                        # disable BDIO upload
--output.include.containerfilesystem=true  # tell DI to output the container file system
--output.path={your output dir}            # tell DI where to put the output
  1. Locate the container file system in the output directory (*.tar.gz) and untar it.
  2. cd into the directory within the untared container file system that you want to scan.
  3. Invoke Detect there.

Excluding files/directories from the returned container file system which excludes them from Detect's signature scan

To exclude certain files and/or directories from the returned file system, you can specify that list of directories with the property --output.containerfilesystem.excluded.paths.

For example, if you are invoking Black Duck Docker Inspector from Detect, and want Detect to exclude the /etc and /usr/bin directories from the signature scan, you could run Detect like this:

./detect.sh --detect.docker.image=ubuntu:latest --detect.docker.passthrough.output.containerfilesystem.excluded.paths=/etc,/usr/bin