Ctrl+K

Installing Supported Kernels (Containers)#

Gateway Provisioners includes tooling to generate kernel specifications that support the following kernels:

  • IPython kernel (Python)

  • Apache Toree (Scala)

  • IRKernel (R)

Kernel specifications reside within the host application image (or volume mount) and their generation is addressed in the applicable sections for Kubernetes and Docker/DockerSwarm. What follows are instructions for how to build the kernel-based image.

Tip

Gateway Provisioners provides a Makefile for building the various kernel images. The instructions that follow will reference make targets to accomplish these tasks.

Kernel-image Dockerfiles#

There are two forms of kernel images that can be created, one for regular or vanilla kernels, another for Spark-based kernels. Both use the same Dockerfile, but Spark-based images will specify a different base image from which the target image is derived.

Spark Base Image#

Spark-based images will be built upon the elyra/gp-spark-base image, whose Dockerfile is located in gateway_provisioners/docker/gp-spark-base. This image builds on jupyter/docker-stacks-foundation:2022-11-15 by installing Spark. However, that base can also be substituted via the build argument BASE_CONTAINER.

To build the spark base image use:

make gp-spark-base

This will create an image named, by default, elyra/gp-spark-base:dev. The organization (elyra) and tag (dev) can be controlled using variables DOCKER_ORG and TAG, respectively. For example, to create a gp-spark-base image with a custom organization and tag of the form my-custom-org/gp-spark-base:my-custom-tag, you can use:

make DOCKER_ORG=my-custom-org TAG=my-custom-tag gp-spark-base

Other arguments that can be specified are SPARK_VERSION (default is 3.3.1), HADOOP_VERSION (2), SCALA_VERSION (2.12), OPENJDK_VERSION (17), and SPARK_CHECKSUM (depends on SPARK_VERSION).

Tip

As with DOCKER_ORG and TAG, each of the above build arguments can be specified on the make command.

Kernel Images#

The Dockerfile used to create a kernel image representing any of the supported kernels is located in gateway_provisioners/docker/kernel-image.

This is a multi-stage build Dockerfile whose build options are driven by docker build arguments. There are three primary arguments that drive a kernel’s image build, BASE_CONTAINER, KERNEL_LANG, and PACKAGE_SOURCE.

BASE_CONTAINER#

Using a BASE_CONTAINER build argument (e.g., --build-arg BASE_CONTAINER=xxx) controls from which image the kernel image will be derived. By default, vanilla kernel images are derived from jupyter/docker-stacks-foundation:2022-11-15 while spark-based kernel images should specify a BASE_CONTAINER of either elyra/gp-spark-base:dev or another applicable image.

Gateway Provisioner’s Makefile supports targets for three spark-based images: gp-kernel-spark-py, gp-kernel-spark-r, and gp-kernel-scala, and two vanilla kernel images: gp-kernel-py and gp-kernel-r. As with the spark base image, the DOCKER_ORG and TAG can be similarly controlled, but by default, the following command will produce images names elyra/gp-kernel-spark-py:dev, elyra/gp-kernel-spark-r:dev, elyra/gp-kernel-scala:dev, elyra/gp-kernel-py:dev, and elyra/gp-kernel-r:dev:

make gp-kernel-spark-py gp-kernel-spark-r gp-kernel-scala gp-kernel-py gp-kernel-r

Attention

The Scala kernel (based on Apache Toree) is predicated on its use within a Spark environment, so a vanilla version of the Scala kernel is not provided.

Tip

The helper-target kernel-images can be used to create all five kernel images via make kernel-images.

KERNEL_LANG#

The second primary build argument to consider is the kernel language. Each of the previously mentioned Makefile targets automatically set --build-arg KERNEL_LANG=<lang> to the expected argument. If not specified, KERNEL_LANG defaults to python. KERNEL_LANG must be one of the following values: python, r, or scala.

It is this build argument that determines which kernel package to install, all of which is handled in the Dockerfile.

The resulting image will include an ENV entry of KERNEL_LANGUAGE with one of the respective values: Python, R, or Scala, as well as a LABEL entry of KERNEL_LANG with the respective lowercase form: python, r, or scala.

PACKAGE_SOURCE#

The third primary build argument is PACKAGE_SOURCE. This identifies from where the Gateway Provisioners installation should come. By default, PACKAGE_SOURCE=release, meaning that the Gateway Provisioners package will be installed from the latest release. If a build argument for PACKAGE_SOURCE specifies local, then the locally built wheel file for Gateway Provisioners will be installed. This option is ideal for development environments.

Because the Makefile is plumbed for PACKAGE_SOURCE, it can be specified directly in the make command. For example, the following will build a vanilla kernel for python using a locally built wheel file:

make PACKAGE_SOURCE=local gp-kernel-py

The resulting image will include a LABEL entry of PACKAGE_SOURCE with one of the respective values: release or local.

Bootstrapping the Kernel Image#

One of the more important tasks performed by kernel-image/Dockerfile is the installation of the bootstrap script. This script becomes the image’s command (CMD) and is placed into the image using the jupyter image-bootstrap install command line script.

CMD /usr/local/bin/bootstrap-kernel.sh

# Install bootstrap and applicable launchers (per languages)
RUN jupyter image-bootstrap install --languages ${KERNEL_LANG}

RUN chown jovyan:users /usr/local/bin/bootstrap-kernel.sh && \
	chmod 0755 /usr/local/bin/bootstrap-kernel.sh && \
	chown -R jovyan:users /usr/local/bin/kernel-launchers

Upon its installation, the CMD is set into the image and the necessary kernel-launcher scripts are placed into /usr/local/bin/kernel-launchers.

On this page
Show Source