Docker Setup

The Best Docker Setup

Setup PostgreSQL on Windows with Docker February 18, 2018 February 18, 2018 / Docker, PostgreSQL / Docker, pgAdmin 4, Postgres, PostgreSQL Over the weekend I finally got the chance to start reading A Curious Moon by Rob Conery which is a book on learning PostgreSQL by following the fictional Dee Yan as she is thrown into database.

It can be used with the Docker Engine 1.8+ on Linux or on Docker for Mac/Windows. This quickstart specifically focuses on using the SQL Server on Linux image. The Windows image is not covered, but you can learn more about it on the mssql-server-windows-developer Docker Hub page.
MongoDB document databases provide high availability and easy scalability.
Run Docker Registry. You have completed setup. You can build registry using docker-compose command. Go to the directory, where we create docker-compose.yml file $ cd myregistry. Now run the following command: $ docker-compose up -d. Docker registry is now up, you can verify the running containers using following command: $ docker ps -a.
Sysctl -w kernel.grsecurity.chrootdenychmod=0 sysctl -w kernel.grsecurity.chrootdenymknod=0 For more information, have a look at the corresponding Github issue. Anyway, this weakening of security is not necessary to do with Alpine 3.4.x and Docker 1.12 as of August 2016 anymore.

TL; DR: An eponymous user per daemon and a shared group with a umask of 002. Consistent path definitions between all containers that maintains the folder structure. Using one volume for Sonarr, Radarr and Lidarr so the download folder and library folder are on the same file system which makes hard links and instant moves possible. And most of all, ignore most of the Docker image’s path documentation!

Note: Many folks find TRaSH's Hardlink Tutorial helpful and easier to understand than this guide. This guide is more conceptual in nature while TRaSH's tutorial walks you through the process.

Introduction

This article will not show you specifics about the best Docker setup, but it describes an overview that you can use to make your own setup the best that it can be. The idea is that you run each docker container as its own user, with a shared group and consistent volumes so every container sees the same path layout. This is easy to say, but difficult to understand and explain.

Multiple users and a shared group

Permissions

Ideally, each software runs as its own user user and they’re all part of a shared group with folder permissions set to 775 (drwxrwxr-x) and files set to 664 (-rw-rw-r--), which is a umask of 002. A sane alternative to this is a single shared user, which would use 755 and 644 which is a umask of 022. You can restrict permissions even more by denying read from “other”, which would be a umask of 007 for a user per daemon or 077 for a single shared user. For a deeper explanation, try the Arch Linux wiki articles about File permissions and attributes and Umask.

UMASK

Many docker images accept a -e UMASK=002 environment variable and some software inside can be configured with a user, group and umask (NZBGet) or folder/file permission (Sonarr/Radarr). This will ensure that files and folders created by one can be read and written by the others. If you’re using existing folders and files, you’ll need to fix their current ownership and permissions too, but going forward they’ll be correct because you set each software up right.

PUID and PGID

Many docker images also take a -e PUID=123 and -e PGID=321 that lets you change the UID/GID used inside to that of an account on the outside. If you ever peak in, you’ll find that username is something like abc, nobody or hotio, but because it uses the UID/GID you pass in, on the outside it looks like the expected user. If you’re using storage from another system via NFS or CIFS, it will make your life easier if that system also has matching users and group. Perhaps let one system pick the UID/GIDs, then re-use those on the other system, assuming they don’t conflict.

Example

You run Sonarr using hotio/sonarr, you’ve created a sonarr user with uid 123 and a shared group media with gid 321 which the sonarr user is a member of. You configure the Docker image to run with -e PUID=123 -e PGID=321 -e UMASK=002. Sonarr also lets you configured user, group as well as folder and file permissions. The previous settings should negate these, but you could configure them if you wanted. Folders would be 775, files 664 and the user/group are a little tricky because inside the container, they have a different name. Maybe abc or nobody. I’d leave all these blank unless you find you need them for some reason.

Single user and optional shared group

Another popular and arguably easier option is a single, shared user. Perhaps even your user. It isn’t as secure and doesn’t follow best practices, but in the end it is easier to understand and implement. The UMASK for this is 022 which results in 755 (drwxr-xr-x) for folders and 644 (-rw-r--r--) for files. The group no longer really matters, so you’ll probably just use the group named after the user. This does make it harder to share with other users, so you may still end up wanting a UMASK of 002 even with this setup.

Ownership and permissions of /config

Don’t forget that your /config volume will also need to have correct ownership and permissions, usually the daemon’s user and that user’s group like sonarr:sonarr and a umask of 022 or 077 so only that user has access. In a single user setup, this would of course be the one user you’ve chosen.

Consistent and well planned paths

The easiest and most important detail is to create unified path definitions across all the containers.

If you’re wondering why hard links aren’t working or why a simple move is taking far longer than it should, this section explains it. The paths you use on the inside matter. Because of how Docker’s volumes work, passing in two volumes such as the commonly suggested /tv and /downloads makes them look like two file systems, even if they aren’t. This means hard links won’t work and instead of an instant move, a slower and more io intensive copy + delete is used. If you have multiple download clients because you’re using torrents and usenet, having a single /downloads path means they’ll be mixed up. Because the Radarr in one container will ask the NZBGet in its own container where files are, using the same path in both means it will all just work. If you don’t, you’d need to fix it with a remote path map.

So pick one path layout and use it for all of them. I’m a fan of /data, but there are other great names like /shared, /media or /dvr. If this can be the same on the outside and inside, your setup will be simpler: one path to remember or if integrating docker and native software. But if not, that’s fine too. For example, Synology might use /Volume1/data and unRAID might use /mnt/user/data on the outside, but /data on the inside is fine.

It is also important to remember that you’ll need to setup or re-configure paths in the software running inside these Docker containers. If you change the paths for your download client, you’ll need to edit its settings to match. If you change your library path, you’ll need to change those settings in Sonarr, Radarr, Lidarr and/or Plex.

Examples

What matters here is the general structure, not the names. You are free to pick folder names that make sense to you. And there are other ways of arranging things too. For example, you’re not likely to download and run into conflicts of identical releases between usenet and torrents, so you could put both in /data/downloads/{movies|music|tv} folders. Downloads don’t even have to be sorted into subfolders either, since movies, music and tv will rarely conflict.

This example data folder has subfolders for torrents and usenet and each of these have subfolders for tv, movie and music downloads to keep things neat. The media folder has nicely named tv, movies and music subfolders, this is your library and what you’d pass to Plex, Kodi, or Emby.

The path for each Docker container can be as specific as needed while still maintaining the correct structure:

Torrents

Torrents only needs access to torrent files, so pass it -v /host/data/torrents:/data/torrents. In the torrent software settings, you’ll need to reconfigure paths and you can sort into subfolders like/data/torrents/{tv|movies|music}.

Usenet

Usenet only needs access to usenet files, so pass it -v /host/data/usenet:/data/usenet. In the usenet software settings, you’ll need to reconfigure paths and you can sort into subfolders like/data/usenet/{tv|movies|music}.

Media Server

Plex/Emby only needs access to your media library, so pass -v /host/data/media:/data/media, which can have any number of subfolders like movies, kids movies, tv, documentary tv and/or music as sub folders.

Sonarr, Radarr and Lidarr

Sonarr, Radarr and Lidarr get everything using -v /host/data:/data because the download folder(s) and media folder will look like and be one file system. Hard links will work and moves will be atomic, instead of copy + delete.

Issues

There are a couple minor issues with not following the Docker image’s suggested paths.

The biggest is that volumes defined in the Dockerfile will get created if they’re not specified, this means they’ll pile up as you delete and re-create the containers. If they end up with data in them, they can consume space unexpectedly and likely in an unsuitable place. You can find a cleanup command in the helpful commands section below. This could also be mitigated by passing in an empty folder for all the volumes you don’t want to use, like /data/empty:/movies and /data/empty:/downloads. Maybe even put a file named DO NOT USE THIS FOLDER inside, to remind yourself.

Another problem is that some images are pre-configured to use the documented volumes, so you’ll need to change settings in the software inside the Docker container. Thankfully, since configuration persists outside the container this is a one time issue. You might also pick a path like /data or /media which some images already define for a specific use. It shouldn’t be a problem, but will be a little more confusing when combined with the previous issues. In the end, it is worth it for working hard links and fast moves. The consistency and simplicity are welcome side effects as well.

If you use the latest version of the abandoned RadarrSync to synchronize two Radarr instances, it depends on mapping the same path inside to a different path on the outside, for example /movies for one instance would point at /data/media/Movies and the other at /data/media/Movies 4k. This breaks everything you’ve read above. There is no good solution, you either use the old version which isn’t as good, do your mapping in a way that is ugly and breaks hard links or just don’t use it at all.

Running containers using

Docker-compose

This is the best option for most users, it lets you control and configure many containers and their interdependence in one file. A good starting place is docker’s own Get started with Docker Compose. You can use composerize or red5d/docker-autocomposeto convert docker run commands into a single docker-compose.yml file.

The below is not a complete working example! The containers only have PID, UID, UMASK and example paths defined to keep it simple.

Update all images and containers

Update individual image and container

Docker Run

Like the Docker Compose example above, the following docker run commands are stripped down to only the PUID, PGID, UMASK and volumes in order to act as an obvious example.

Systemd

I don’t run a full Docker setup, so I manage my few Docker containers with individual systemd service files. It standardizes control and makes dependencies simpler for both native and docker services. The generic example below can be adapted to any container by adjusting or adding the various values and options.

Helpful commands

List running containers

Shell inside a container

For more information, see the docker exec documentation.

Prune docker

Remove unused containers, networks, volumes, images and build cache. As the WARNING this command gives says, this will remove all of the previously mentioned items for anything not in use by a running container. In a correctly configured environment, this is fine. But be aware and proceed cautiously the first time. See the docker system prune documentation for more details.

Get docker run command

Getting the docker run command from GUI managers can be hard, this docker image makes it easy for a running container (source).

Get docker-compose

Additionally, you may check out TRaSH's Guide for docker-compose

Getting a docker-compose.yml from running instances is possible with red5d/docker-autocompose, in case you’ve already started your containers with docker run or docker create and want to change to docker-compose style. It is also great for sharing your settings with others, since it doesn’t matter what management software you’re using. The last argument(s) are your container names and you can pass in as many as needed at the same time. The first container name is required, more are optional. You can see container names in the NAMES column of `docker ps`, they're usually set by you or might be generated based on the image like binhex-qbittorrent. It is not the image name, like binhex/arch-qbittorrentvpn.

Troubleshoot networking

Most Docker images don’t have many useful tools in them for troubleshooting, but you can attach a network troubleshooting type image to an existing container to help with that.

Recursively chown user and group

Recursively chmod to 775/664

Find UID/GID for user

Examine files for hard links

Interesting docker images

rasmunk/sshfs let you create an sshfs volume, perfect for a seedbox setup using a remote mount instead of sync. Better documentation, including examples can be found at the github rasmunk/docker-volume-sshfs repository. This is a more recently maintained fork of vieux/sshfs.
hotio’ssonarr, radarr and lidarr images let you run the built in version or specify an alternative via environment variable. The documentation and Dockerfile also don’t make any poor path suggestions.
hotio’sombi, jackett, nzbhydra2 and bazarr are useful too, but don’t really require any special permissions or paths.
hotio’sunpackerr is useful for packed torrent extraction across a variety of torrent clients where unpacking is lacking or missing entirely.
binhex’sqbittorrent, deluge and rtorrent are popular torrent clients with built in VPN support. For usenet, there is sabnzbd and nzbget.
binhex’ssonarr, radarr and lidarr images suggest default paths that don’t allow for hard linking, instead follow the process described above and pass in a single volume.
linuxserver.io’s images also suggest default paths that don’t allow for hard linking, instead follow the process described above and pass in a single volume. But they do have images for a lot of software and they’re well maintained.
pyouroboros/ouroboros or containrrr/watchtower automatically update your running Docker containers to the latest available image. These are not recommended if you use Docker Compose.

Custom Docker Network and DNS

One interesting feature of a custom Docker network is that it gets its own DNS server. If you create a bridge network for your containers, you can use their hostnames in your configuration. For example, if you docker run --network=isolated --hostname=deluge binhex/arch-deluge and docker run --network=isolated --hostname=radarr binhex/arch-radarr, you can then configure the Download Client in Radarr to point at just deluge and it’ll work and communicate on its own private network. Which means if you wanted to be even more secure, you could stop forwarding that port too. If you put your reverse proxy container on the same network, you can even stop forwarding the web interface ports and make them even more secure.

Common Problems

Correct outside paths, incorrect inside paths

Many people read this and think they understand, but they end up seeing the outside path correctly to something like /data/usenet, but then they miss the point and set the inside path to /downloads still.

Running Docker containers as root or changing users around

If you find yourself running your containers as root:root, you’re doing something wrong. If you’re not passing in a UID and GID, you’ll be using whatever the default is for the image and that will be unlikely to line up with a reasonable user on your system. And if you’re changing the user and group your Docker containers are running as, you’ll probably end up with permissions issues on folders like the /config folder which will likely have files and folders in them that got created the first time with the UID/GID you used the first time.

Running Docker containers with umask 000

If you find yourself setting a UMASK of 000 (which is 777 for folders and 666 for files), you’re also doing something wrong. It leaves your files and folders read/write to everyone, which is poor Linux hygiene.

LinuxServer.io

The entire reason this article even exists is linuxserver.io’s documentation and default path suggestions. They’re singlehandedly responsible for the /tv, /movies and /downloads paths which break hard links and atomic moves. It is for this reason that none of them are specifically recommended by this article. That said, once you understand the problem with those paths and that you don’t have to use them, their images are fine. So feel free to use them and most of their documentation. Just ignore their path suggestions.

Getting Help

Need some help? For real time, chat style support try the Sonarr or Radarr Discord servers. If you prefer forum style support, make a post in /r/sonarr or /r/radarr.

Retrieved from 'https://wiki.servarr.com/index.php?title=Docker_Guide&oldid=3327'

-->

Applies to: SQL Server (all supported versions) - Linux

Note

The examples shown below use the docker.exe but most of these commands also work with Podman. It provides the CLI similar to Docker container Engine. You can read more about podman here.

In this quickstart, you use Docker to pull and run the SQL Server 2017 container image, mssql-server-linux. Then connect with sqlcmd to create your first database and run queries.

Tip

If you want to run SQL Server 2019 containers, see the SQL Server 2019 version of this article.

Note

Starting with SQL Server 2019 CU3, Ubuntu 18.04 is supported.

In this quickstart, you use Docker to pull and run the SQL Server 2019 container image, mssql-server. Then connect with sqlcmd to create your first database and run queries.

Tip

This quickstart creates SQL Server 2019 containers. If you prefer to create SQL Server 2017 containers, see the SQL Server 2017 version of this article.

This image consists of SQL Server running on Linux based on Ubuntu 18.04. It can be used with the Docker Engine 1.8+ on Linux or on Docker for Mac/Windows. This quickstart specifically focuses on using the SQL Server on Linux image. The Windows image is not covered, but you can learn more about it on the mssql-server-windows-developer Docker Hub page.

Prerequisites

Docker Engine 1.8+ on any supported Linux distribution or Docker for Mac/Windows. For more information, see Install Docker.
Docker overlay2 storage driver. This is the default for most users. If you find that you are not using this storage provider and need to change, see the instructions and warnings in the docker documentation for configuring overlay2.
Minimum of 2 GB of disk space.
Minimum of 2 GB of RAM.
System requirements for SQL Server on Linux.

Pull and run the 2017 container image

Before starting the following steps, make sure that you have selected your preferred shell (bash, PowerShell, or cmd) at the top of this article.

Pull the SQL Server 2017 Linux container image from Microsoft Container Registry.
Tip
If you want to run SQL Server 2019 containers, see the SQL Server 2019 version of this article.
The previous command pulls the latest SQL Server 2017 container image. If you want to pull a specific image, you add a colon and the tag name (for example, mcr.microsoft.com/mssql/server:2017-GA-ubuntu). To see all available images, see the mssql-server Docker hub page.
For the bash commands in this article, sudo is used. On macOS, sudo might not be required. On Linux, if you do not want to use sudo to run Docker, you can configure a docker group and add users to that group. For more information, see Post-installation steps for Linux.

To run the container image with Docker, you can use the following command from a bash shell (Linux/macOS) or elevated PowerShell command prompt.

Note

If you are using PowerShell Core, replace the double quotes with single quotes.

Note

The password should follow the SQL Server default password policy, otherwise the container can not setup SQL server and will stop working. By default, the password must be at least 8 characters long and contain characters from three of the following four sets: Uppercase letters, Lowercase letters, Base 10 digits, and Symbols. You can examine the error log by executing the docker logs command.

By default, this creates a container with the Developer edition of SQL Server 2017. The process for running production editions in containers is slightly different. For more information, see Run production container images.

The following table provides a description of the parameters in the previous docker run example:

Parameter	Description
-e 'ACCEPT_EULA=Y'	Set the ACCEPT_EULA variable to any value to confirm your acceptance of the End-User Licensing Agreement. Required setting for the SQL Server image.
-e 'SA_PASSWORD=<YourStrong@Passw0rd>'	Specify your own strong password that is at least 8 characters and meets the SQL Server password requirements. Required setting for the SQL Server image.
-p 1433:1433	Map a TCP port on the host environment (first value) with a TCP port in the container (second value). In this example, SQL Server is listening on TCP 1433 in the container and this is exposed to the port, 1433, on the host.
--name sql1	Specify a custom name for the container rather than a randomly generated one. If you run more than one container, you cannot reuse this same name.
-h sql1	Used to explicitly set the container hostname, if you don't specify it, it defaults to the container ID which is a randomly generated system GUID.
-d	Run the container in the background (daemon)
mcr.microsoft.com/mssql/server:2017-latest	The SQL Server 2017 Linux container image.

To view your Docker containers, use the docker ps command.
You should see output similar to the following screenshot:
If the STATUS column shows a status of Up, then SQL Server is running in the container and listening on the port specified in the PORTS column. If the STATUS column for your SQL Server container shows Exited, see the Troubleshooting section of the configuration guide.

The -h (host name) parameter as discussed above, changes the internal name of the container to a custom value. This is the name you'll see returned in the following Transact-SQL query:

Setting -h and --name to the same value is a good way to easily identify the target container.

As a final step, change your SA password because the SA_PASSWORD is visible in ps -eax output and stored in the environment variable of the same name. See steps below.

Pull and run the 2019 container image

Before starting the following steps, make sure that you have selected your preferred shell (bash, PowerShell, or cmd) at the top of this article.

Pull the SQL Server 2019 Linux container image from Microsoft Container Registry.
Note
If you are using PowerShell Core, replace the double quotes with single quotes.
Tip
This quickstart uses the SQL Server 2019 Docker image. If you want to run the SQL Server 2017 image, see the SQL Server 2017 version of this article.
The previous command pulls the SQL Server 2019 container image based on Ubuntu. To instead use container images based on RedHat, see Run RHEL-based container images. To see all available images, see the mssql-server-linux Docker hub page.
For the bash commands in this article, sudo is used. On macOS, sudo might not be required. On Linux, if you do not want to use sudo to run Docker, you can configure a docker group and add users to that group. For more information, see Post-installation steps for Linux.

To run the container image with Docker, you can use the following command from a bash shell (Linux/macOS) or elevated PowerShell command prompt.

Note

By default, this creates a container with the Developer edition of SQL Server 2019.

The following table provides a description of the parameters in the previous docker run example:

Parameter	Description
-e 'ACCEPT_EULA=Y'	Set the ACCEPT_EULA variable to any value to confirm your acceptance of the End-User Licensing Agreement. Required setting for the SQL Server image.
-e 'SA_PASSWORD=<YourStrong@Passw0rd>'	Specify your own strong password that is at least 8 characters and meets the SQL Server password requirements. Required setting for the SQL Server image.
-p 1433:1433	Map a TCP port on the host environment (first value) with a TCP port in the container (second value). In this example, SQL Server is listening on TCP 1433 in the container and this is exposed to the port, 1433, on the host.
--name sql1	Specify a custom name for the container rather than a randomly generated one. If you run more than one container, you cannot reuse this same name.
-h sql1	Used to explicitly set the container hostname, if you don't specify it, it defaults to the container ID which is a randomly generated system GUID.
mcr.microsoft.com/mssql/server:2019-latest	The SQL Server 2019 Ubuntu Linux container image.

To view your Docker containers, use the docker ps command.
You should see output similar to the following screenshot:
If the STATUS column shows a status of Up, then SQL Server is running in the container and listening on the port specified in the PORTS column. If the STATUS column for your SQL Server container shows Exited, see Troubleshooting SQL Server Docker containers.

The -h (host name) parameter as discussed above, changes the internal name of the container to a custom value. This changes the internal name of the container to a custom value. This is the name you'll see returned in the following Transact-SQL query:

Setting -h and --name to the same value is a good way to easily identify the target container.

As a final step, change your SA password because the SA_PASSWORD is visible in ps -eax output and stored in the environment variable of the same name. See steps below.

Change the SA password

The SA account is a system administrator on the SQL Server instance that gets created during setup. After creating your SQL Server container, the SA_PASSWORD environment variable you specified is discoverable by running echo $SA_PASSWORD in the container. For security purposes, change your SA password.

Choose a strong password to use for the SA user.
Use docker exec to run sqlcmd to change the password using Transact-SQL. In the following example, replace the old password, <YourStrong!Passw0rd>, and the new password, <YourNewStrong!Passw0rd>, with your own password values.

Connect to SQL Server

The following steps use the SQL Server command-line tool, sqlcmd, inside the container to connect to SQL Server.

Use the docker exec -it command to start an interactive bash shell inside your running container. In the following example sql1 is name specified by the --name parameter when you created the container.
Once inside the container, connect locally with sqlcmd. Sqlcmd is not in the path by default, so you have to specify the full path.
Tip
You can omit the password on the command-line to be prompted to enter it.
If successful, you should get to a sqlcmd command prompt: 1>.

Create and query data

The following sections walk you through using sqlcmd and Transact-SQL to create a new database, add data, and run a query.

Create a new database

The following steps create a new database named TestDB.

From the sqlcmd command prompt, paste the following Transact-SQL command to create a test database:
On the next line, write a query to return the name of all of the databases on your server:
The previous two commands were not executed immediately. Type GO on a new line to execute the previous commands:

Docker Setup Wsl2

Insert data

Next create a new table, Inventory, and insert two new rows.

From the sqlcmd command prompt, switch context to the new TestDB database:
Create new table named Inventory:
Insert data into the new table:
Type GO to execute the previous commands:

Select data

Now, run a query to return data from the Inventory table.

From the sqlcmd command prompt, enter a query that returns rows from the Inventory table where the quantity is greater than 152:
Execute the command:

Exit the sqlcmd command prompt

To end your sqlcmd session, type QUIT:
To exit the interactive command-prompt in your container, type exit. Your container continues to run after you exit the interactive bash shell.

Connect from outside the container

You can also connect to the SQL Server instance on your Docker machine from any external Linux, Windows, or macOS tool that supports SQL connections.

The following steps use sqlcmd outside of your container to connect to SQL Server running in the container. These steps assume that you already have the SQL Server command-line tools installed outside of your container. The same principles apply when using other tools, but the process of connecting is unique to each tool.

Find the IP address for the machine that hosts your container. On Linux, use ifconfig or ip addr. On Windows, use ipconfig.
For this example, install the sqlcmd tool on your client machine. For more information, see Install sqlcmd on Windows or Install sqlcmd on Linux.
Run sqlcmd specifying the IP address and the port mapped to port 1433 in your container. In this example, that is the same port, 1433, on the host machine. If you specified a different mapped port on the host machine, you would use it here. You will also need to open the appropriate inbound port on your firewall to allow the connection.
Run Transact-SQL commands. When finished, type QUIT.

Other common tools to connect to SQL Server include:

Remove your container

If you want to remove the SQL Server container used in this tutorial, run the following commands:

Warning

Stopping and removing a container permanently deletes any SQL Server data in the container. If you need to preserve your data, create and copy a backup file out of the container or use a container data persistence technique.

Docker Setup For Windows 10

Docker demo

After you have tried using the SQL Server container image for Docker, you might want to know how Docker is used to improve development and testing. The following video shows how Docker can be used in a continuous integration and deployment scenario.

Docker Setup Ubuntu

Next steps

Docker Setup

For a tutorial on how to restore database backup files into a container, see Restore a SQL Server database in a Linux Docker container. Explore other scenarios, such as running multiple containers, data persistence, and troubleshooting.

Docker Setup Guide

Also, check out the mssql-docker GitHub repository for resources, feedback, and known issues.