Summary
Building our Django app locally (i.e. no Docker container wrapping it) works great. Building the same app in Docker fails. Hint: make sure you know which requirements.txt file you’re using to build the app. (And get familiar with the -f parameter for Docker commands.)
Problem
When I first started build the Docker container, I was getting the ImportError error after the container successfully builds:
ImportError: No module named 'rest_framework_swagger'
Research
The only half-useful hit on StackOverflow was this one, and it didn’t seem like it explicitly addressed my issue in Docker:
…And The Lightning Bolt Struck
However, with enough time and desperation I finally understood that that article wasn’t wrong either. I wasn’t using the /requirements.txt that contained all the dependencies – I was using the incomplete/abandoned /budget_proj/requirements.txt file, which lacked a key dependency.
Aside
I wasn’t watching the results of pip install closely enough – and when running Docker-compose up --build multiple times, the layer of interest won’t rebuild if there’s no changes to that layer’s inputs. (Plus this is a case where there’s no error message thrown, just one or two fewer pip installs – and who notices that until they’ve spent the better part of two days on the problem?)
Detailed Diagnostics
If you look closely at our project from that time, you’ll notice there are actually two copies of requirements.txt – one at the repo root and one in the /budget_proj/ folder.
Developers who are just testing Django locally will simply launch pip install -r requirements.txt from the root directory of their clone of the repo. This is fine and good. This is the result of the pip install -r requirements.txt when using the expected file:
$ pip install -r requirements.txt Collecting appdirs==1.4.0 (from -r requirements.txt (line 1)) Using cached appdirs-1.4.0-py2.py3-none-any.whl Collecting Django==1.10.5 (from -r requirements.txt (line 2)) Using cached Django-1.10.5-py2.py3-none-any.whl Collecting django-filter==1.0.1 (from -r requirements.txt (line 3)) Using cached django_filter-1.0.1-py2.py3-none-any.whl Collecting django-rest-swagger==2.1.1 (from -r requirements.txt (line 4)) Using cached django_rest_swagger-2.1.1-py2.py3-none-any.whl Collecting djangorestframework==3.5.4 (from -r requirements.txt (line 5)) Using cached djangorestframework-3.5.4-py2.py3-none-any.whl Requirement already satisfied: packaging==16.8 in ./budget_venv/lib/python3.5/site-packages (from -r requirements.txt (line 6)) Collecting psycopg2==2.7 (from -r requirements.txt (line 7)) Using cached psycopg2-2.7-cp35-cp35m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl Collecting pyparsing==2.1.10 (from -r requirements.txt (line 8)) Using cached pyparsing-2.1.10-py2.py3-none-any.whl Collecting requests==2.13.0 (from -r requirements.txt (line 9)) Using cached requests-2.13.0-py2.py3-none-any.whl Requirement already satisfied: six==1.10.0 in ./budget_venv/lib/python3.5/site-packages (from -r requirements.txt (line 10)) Collecting gunicorn (from -r requirements.txt (line 12)) Using cached gunicorn-19.7.0-py2.py3-none-any.whl Collecting openapi-codec>=1.2.1 (from django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Collecting coreapi>=2.1.1 (from django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Collecting simplejson (from django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Using cached simplejson-3.10.0-cp35-cp35m-macosx_10_11_x86_64.whl Collecting uritemplate (from coreapi>=2.1.1->django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Using cached uritemplate-3.0.0-py2.py3-none-any.whl Collecting coreschema (from coreapi>=2.1.1->django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Collecting itypes (from coreapi>=2.1.1->django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Collecting jinja2 (from coreschema->coreapi>=2.1.1->django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Using cached Jinja2-2.9.5-py2.py3-none-any.whl Collecting MarkupSafe>=0.23 (from jinja2->coreschema->coreapi>=2.1.1->django-rest-swagger==2.1.1->-r requirements.txt (line 4)) Installing collected packages: appdirs, Django, django-filter, uritemplate, requests, MarkupSafe, jinja2, coreschema, itypes, coreapi, openapi-codec, simplejson, djangorestframework, django-rest-swagger, psycopg2, pyparsing, gunicorn Found existing installation: appdirs 1.4.3 Uninstalling appdirs-1.4.3: Successfully uninstalled appdirs-1.4.3 Found existing installation: pyparsing 2.2.0 Uninstalling pyparsing-2.2.0: Successfully uninstalled pyparsing-2.2.0 Successfully installed Django-1.10.5 MarkupSafe-1.0 appdirs-1.4.0 coreapi-2.3.0 coreschema-0.0.4 django-filter-1.0.1 django-rest-swagger-2.1.1 djangorestframework-3.5.4 gunicorn-19.7.0 itypes-1.1.0 jinja2-2.9.5 openapi-codec-1.3.1 psycopg2-2.7 pyparsing-2.1.10 requests-2.13.0 simplejson-3.10.0 uritemplate-3.0.0
However, because our Django application (and the related Docker files) is contained in a subdirectory off the repo root (i.e. in the /budget_proj/ folder) – and because I was an idiot at the time and didn’t know about the -f parameter for docker-compose , so I was convinced I had to run docker-compose from the same directory as docker-compose.yml – docker-compose didn’t have access to files in the parent directory of wherever it was launched. Apparently Docker effectively “chroots” its commands so it doesn’t have access to ../bin/requirements.txt for example.
So when docker-compose launched pip install -r requirements.txt, it could only access this one and gives us this result instead:
Step 12/12 : WORKDIR /code ---> 8626fa515a0a Removing intermediate container 05badf699f66 Successfully built 8626fa515a0a Recreating budgetproj_budget-service_1 Attaching to budgetproj_budget-service_1 web_1 | Running docker-entrypoint.sh... web_1 | [2017-03-16 00:31:34 +0000] [5] [INFO] Starting gunicorn 19.7.0 web_1 | [2017-03-16 00:31:34 +0000] [5] [INFO] Listening at: http://0.0.0.0:8000 (5) web_1 | [2017-03-16 00:31:34 +0000] [5] [INFO] Using worker: sync web_1 | [2017-03-16 00:31:34 +0000] [8] [INFO] Booting worker with pid: 8 web_1 | [2017-03-16 00:31:35 +0000] [8] [ERROR] Exception in worker process web_1 | Traceback (most recent call last): web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker web_1 | worker.init_process() web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process web_1 | self.load_wsgi() web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi web_1 | self.wsgi = self.app.wsgi() web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi web_1 | self.callable = self.load() web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load web_1 | return self.load_wsgiapp() web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp web_1 | return util.import_app(self.app_uri) web_1 | File "/usr/local/lib/python3.5/site-packages/gunicorn/util.py", line 376, in import_app web_1 | __import__(module) web_1 | File "/code/budget_proj/wsgi.py", line 16, in <module> web_1 | application = get_wsgi_application() web_1 | File "/usr/local/lib/python3.5/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application web_1 | django.setup(set_prefix=False) web_1 | File "/usr/local/lib/python3.5/site-packages/django/__init__.py", line 27, in setup web_1 | apps.populate(settings.INSTALLED_APPS) web_1 | File "/usr/local/lib/python3.5/site-packages/django/apps/registry.py", line 85, in populate web_1 | app_config = AppConfig.create(entry) web_1 | File "/usr/local/lib/python3.5/site-packages/django/apps/config.py", line 90, in create web_1 | module = import_module(entry) web_1 | File "/usr/local/lib/python3.5/importlib/__init__.py", line 126, in import_module web_1 | return _bootstrap._gcd_import(name[level:], package, level) web_1 | ImportError: No module named 'rest_framework_swagger' web_1 | [2017-03-16 00:31:35 +0000] [8] [INFO] Worker exiting (pid: 8) web_1 | [2017-03-16 00:31:35 +0000] [5] [INFO] Shutting down: Master web_1 | [2017-03-16 00:31:35 +0000] [5] [INFO] Reason: Worker failed to boot. budgetproj_web_1 exited with code 3
Coda
It has been pointed out that not only is it redundant for the project to have two requirements.txt files (I agree, and when we find the poor soul who inadvertently added the second file, they’ll be sacked…from our volunteer project ;)…
…but also that if we’re encapsulating our project’s core application in a subdirectory (called budget_proj), then logically that is where the “legit” requirements.txt file belongs – not at the project’s root, just because that’s where you normally find requirements.txt in a repo.
The rattled cough of a Hyper-Empathetic PM 