42
loading...
Understand Django: Making Sense Of Settings
In the last Understand Django article, we looked at a storage concept in Django called sessions. Sessions help us answer questions like "How does Django know when a user is logged in?" or "Where can the framework store data for a visitor on your app?"
With this article, you'll learn about Django settings and how to manage the configuration of your application. We'll also look at tools to help you to be extra effective with settings.
To run properly, Django needs to be configured. We need to understand where this configuration comes from. Django has the ability to use default configuration values or values set by developers like yourself, but where does it get those from?
Early in the process of starting a Django application, Django will internally import the following:
from django.conf import settingsThis
settings import is a module level object created in django/conf/__init__.py. The settings object has attributes added to it from two primary sources.The first source is a set of global default settings that come from the framework. These global settings are from
django/conf/global_settings.py and provide a set of initial values for configuration that Django needs to operate.The second source of configuration settings comes from user defined values. Django will accept a Python module and apply its module level attributes to the
settings object. To find the user module, Django searches for a DJANGO_SETTINGS_MODULE environment variable.Environment variables are not a Django concept. When any program runs on a computer, the operating system makes certain data available to the running program. This set of data is called the program's "environment," and each piece of data in that set is an environment variable.
If you're starting Django from a terminal, you can view the environment variables that Django will receive from the operating system by running the
env command on macOS or Linux, or the set command on Windows.We can add our own environment variables to the environment with the
export command on macOS or Linux, or the set command on Windows. Environment variables are typically named in all capital letters.$ export HELLO=worldNow that we have a base understanding of environment variables, let's return to the
DJANGO_SETTINGS_MODULE variable. The variable's value should be the location of a Python module containing any settings that a developer wants to change from Django's default values.If you create a Django project with
startproject and use project as the name, then you will find a generated file called project/settings.py in the output. When Django runs, you could explicitly instruct Django with:$ export DJANGO_SETTINGS_MODULE=project.settingsInstead of supplying the file path, the
DJANGO_SETTINGS_MODULE should be in a Python module dotted notation.You may not actually need to set
DJANGO_SETTINGS_MODULE explicitly. If you stick with the same settings file that is created by startproject, you can find a line in wsgi.py that looks like:os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')Because of this line, Django will attempt to read from
project.settings (or whatever you named your project) without the need to explicitly set DJANGO_SETTINGS_MODULE.Once Django reads the global settings and any user defined settings, we can get any configuration from the
settings object via attribute access. This convention of keeping all configuration in the settings object is a convenient pattern that the framework, third party library ecosystem, and you can depend on.$ ./manage.py shell
>>> from django.conf import settings
>>> settings.SECRET_KEY
'a secret to everybody'The
settings object is a shared item so it is generally thought to be a Really Bad Idea™ to edit and assign to the object directly. Keep your settings in your settings module!That's the core of Django configuration. We're ready to focus in on the user defined settings and our responsibilities as Django app developers.
There are multiple ways to deal with settings modules and how to populate those modules with the appropriate values for different environments. Let's look at some popular patterns.
A Django settings module is a Python module. Nothing is stopping us from using the full power of Python to configure that module the way we want.
Minimally, you will probably have at least two environments where your Django app runs:
We should know by now that setting
DEBUG = True is a terrible idea for a live Django site, so how can we get the benefits of the debug mode without having DEBUG set to True in our module?One technique is to use separate settings modules. With this strategy, you can pick which environment your Django app should run for by switching the
DJANGO_SETTINGS_MODULE value to pick a different environment. You might have modules like:project.settings.devproject.settings.stageproject.settings.productionThese examples would be for a local development environment on your laptop, a staging environment (which is a commonly used pattern for testing a site that is as similar to the live site as possible without being the live site), and a production environment. As a reminder from the deployment article, the software industry like to call the primary site for customers "production."
This strategy has certain challenges to consider. Should you replicate settings in each file or use some common module between them?
If you decide to replicate the settings across modules, you'll have the advantage that the settings module shows all of the settings in a single place for that environment. The disadvantage is that keeping the common settings the same could be a challenge if you forget to update one of the modules.
On the other hand, you could use a common module. The advantage to this form is that the common settings can be in a single location. The environment specific files only need to record the differences between the environments. The disadvantage is that it is harder to get a clear picture of all the settings of that environment.
If you decide to use a common module, this style is often implemented with a
* import. I can probably count on one hand the number of places where I'm ok with a * import, and this is one of them. In most cases the Python community prefers explicit over implicit, and the idea extends to the treatment of imports. Explicit imports make it clear what a module is actually using. The * import is very implicit, and it makes it unclear what a module uses. For the case of a common settings module, a * import is actually positive because we want to use everything in the common module.Let's make this more concrete. Assume that you have a
project.settings.base module. This module would hold your common settings for your app. I'd recommend that you try to make your settings safe and secure by default. For instance, use DEBUG = False in the base settings and force other settings modules to opt-in to the more unsafe behavior.For your local development environment on your laptop, you could use
project.settings.dev. This settings module would look like:# project/settings/dev.py
from project.settings.base import *
DEBUG = True
# Define any other settings that you want to override.
...By using the
* import in the dev.py file, all the settings from base.py are pulled into the module level scope. Where you want a setting to be different, you set the value in dev.py. When Django starts using DJANGO_SETTINGS_MODULE of project.settings.dev, all the values from base.py will be used via dev.py.This scheme gives you control to define common things once, but there is still a big challenge with this. What do we do about settings that need to be kept secret (e.g., API keys)?
Don't commit secret data to your code repository! Adding secrets to your source control tool like Git is usually not a good idea. This is especially true if you have a public repository on GitHub. Think no one is paying attention to your repo? Think again! There are tools out there that scan every public commit made to GitHub. These tools are specifically looking for secret data to exploit.
If you can't safely add secrets to your code repo, where can we add them instead? You can use environment variables! Let's look at another scheme for managing settings with environment variables.
In Python, you can access environment variables through the
os module. The module contains the environ attribute, which functions like a dictionary.By using environment variables, your settings module can get configuration settings from the external environment that is running the Django app. This is a solid pattern because it can accomplish two things:
Here's an example of secret data management:
# project/settings.py
import os
SECRET_KEY = os.environ['SECRET_KEY']
...Django needs a secret key for a variety of safe hashing purposes. There is a warning in the default
startproject output that reads:# SECURITY WARNING: keep the secret key used in production secret!By moving the secret key value to an environment variable that happens to have a matching name of
SECRET_KEY, we won't be committing the value to source control for some nefarious actor to discover.This pattern works really well for secrets, but it can also work well for any configuration that we want to vary between environments.
For instance, on one of my projects, I use the excellent Anymail package to send emails via an email service provider (of the ESPs, I happen to use SendGrid). When I'm working with my development environment, I don't want to send real email. Because of that, I use an environment variable to set Django's
EMAIL_BACKEND setting. This let's me switch between the Anymail backend and Django's built-in django.core.mail.backends.console.EmailBackend that prints emails to the terminal instead.If I did this email configuration with
os.environ, it would look like:# project/settings.py
import os
EMAIL_BACKEND = os.environ.get(
'EMAIL_BACKEND', "anymail.backends.sendgrid.EmailBackend")
...I prefer to make my default settings closer to the live site context. This not only leads to safer behavior (because I have to explicitly opt-out of safer settings like switching to
DEBUG = False), but it also means that my live site has less to configure. That's good because there are fewer chances to make configuration mistakes on the site that matters most: the one where my customers are.We need to be aware of a big gotcha with using environment variables. Environment variables are only available as a
str type. This is something to be aware because there will be times when you want a boolean settings value or some other type of data. In a situation where you need a different type, you have to coerce a str into the type you need. In other words, don't forget that every string except the empty string is truthy in Python:>>> not_false = "False"
>>> bool(not_false)
TrueIn the next section, we will see tools that help alleviate this typing problem.
Note: As you learn more about settings, you will probably encounter advice that says to avoid using environment variables. This is well intentioned advice that highlights that there is some risk with using environment variables. With this kind of advice, you may read a recommendation for secrets management tools like HashiCorp Vault. These are good tools, but consider them a more advanced topic. In my opinion, using environment variables for secrets management is a reasonably low risk storage mechanism.
We can focus on two categories of tools that can help you manage your settings in Django: built-in tools and third party libraries.
The built-in tool that is available to you is the
diffsettings command. This tool makes it easy to see the computed settings of your module. Since settings can come from multiple files (including Django's global_settings.py) or environment variables, inspecting the settings output of diffsettings is more convenient than thinking through how a setting is set.By default,
diffsettings will show a comparison of the settings module to the default Django settings. Settings that aren't in the defaults are marked with ### after the value to indicate that they are different.I find that the default output is not the most useful mode. Instead, you can instruct
diffsettings to output in a "unified" format. This format looks a lot more like a code diff. In addition, Django will colorize that output so that it's easier to see. Here's an example of some of the security settings by running ./manage.py diffsettings --output unified for one of my projects.- SECURE_HSTS_INCLUDE_SUBDOMAINS = False
+ SECURE_HSTS_INCLUDE_SUBDOMAINS = True
- SECURE_PROXY_SSL_HEADER = None
+ SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')Finally, I'll note that you can actually compare two separate settings modules. Let's say you wanted to compare settings between your development mode and your live site. Assuming your settings files are names like I described earlier, you could run something like:
$ ./manage.py diffsettings \
--default project.settings.dev \
--settings project.settings.prod \
--output unifiedBy using the
--default flag, we instruct Django that project.settings.dev is the baseline for comparison. This version of the command will show where the two settings modules are different.Django only includes this single tool for working with settings, but I hope you can see that it's really handy. Now let's talk about a useful third party library that can help you with settings.
Earlier in the article, I noted that dealing with environment variables has the pitfall of working with string data for everything. Thankfully, there is a package that can help you work with environment variables. The project is called django-environ. django-environ primarily does two important things that I value:
What does type coercion look like? With
django-environ, you start with Env object.# project/settings.py
import environ
env = environ.Env()The keyword arguments to
Env describe the different environment variables that you expect the app to process. The key is the name of the environment variable. The value is a two element tuple. The first tuple element is the type you want, and the second element is a default value if the environment variable doesn't exist.If you want to be able to control
DEBUG from an environment variable, the settings would be:# project/settings.py
import environ
env = environ.Env(
DEBUG=(bool, False),
)
DEBUG = env("DEBUG")With this setup, your app will be safe by default with
DEBUG set to False, but you'll be able to override that via the environment. django-environ works with a handful of strings that it will accept as True such as "on", "yes", "true", and others (see the documentation for more details).Once you start using environment variables, you'll want an convenient way to set them when your app runs. Manually calling
export for all your variables before running your app is a totally unsustainable way to run apps.The
Env class comes with a handy class method named read_env. With this method, your app can read environment variables into os.environ from a file. Conventionally, this file is named .env, and the file contains a list of key/value pairs that you want as environment variables. Following our earlier example, here's how we could set our app to be in debug mode:# .env
DEBUG=onBack in the settings file, you'd include
read_env:# project/settings.py
import environ
environ.Env.read_env()
env = environ.Env(
DEBUG=(bool, False),
)
DEBUG = env("DEBUG")If you use a
.env file, you will occasionally find a need to put secrets into this file for testing. Since the file can be a source for secrets, you should add this to .gitignore or ignore it in whatever version control system you use. As time goes on, the list of variables and settings will likely grow, so it's also a common pattern to create a .env.example file that you can use as a template in case you ever need to start with a fresh clone of your repository.Now we've looked at multiple strategies and tools for managing settings. I've used many of these schemes on various Django projects, so what is my preferred setup?
For the majority of uses, I find that working with
django-environ in a single file is the best pattern in my experience.When I use this approach, I make sure that all of my settings favor a safe default configuration. This minimizes the configuration that I have to do for a live site.
I like the flexibility of the pattern, and I find that I can quickly set certain configurations when developing. For instance, when I want to do certain kinds of testing like checking email rendering, I'll call something like:
$ EMAIL_TESTING=on ./manage.py runserverMy settings file has a small amount of configuration to alter the email settings to point emails to a local SMTP server tool called MailHog. Because I set an environment variable directly on my command line call, I can easily switch into a mode that sends email to MailHog for quick review.
Overall, I like the environment variable approach, but I do use more than one settings file for one important scenario: testing.
When I run my unit tests, I want to guarantee that certain conditions are always true. There are things that a test suite should never do in the vast majority of cases. Sending real emails is a good example. If I happen to configure my
.env to test real emails for the local environment, I don't want my tests to send out an emails accidentally.Thus, I create a separate testing settings file and configure my test runner (pytest) to use those settings. This settings file does mostly use the base environment, but I'll override some settings with explicit values. Here's how I protect myself from accidental live emails:
# project/testing_settings.py
from .settings import *
# Make sure that tests are never sending real emails.
EMAIL_BACKEND = "django.core.mail.backends.locmem.EmailBackend"Even though my
Env will look for an EMAIL_BACKEND environment variable to configure that setting dynamically, the testing setting is hardcoded to make email sending accidents impossible.The combination of a single file for most settings sprinkled with a testing settings file for safety is the approach that has worked the best for me.
In this article, you learned about Django settings and how to manage the configuration of your application. We covered:
In the next article, we will look at how to handle files and media provided by users (e.g., profile pictures). You'll learn about:
If you'd like to follow along with the series, please feel free to sign up for my newsletter where I announce all of my new content. If you have other questions, you can reach me online on Twitter where I am @mblayman.
42