Python directory structure and imports
Like the articles? Buy the book! Dead Simple Python by Jason C. McDonald is available from No Starch Press. Show
The worst part of tutorials is always their simplicity, isn't it? Rarely will you find one with more than one file, far more seldom with multiple directories. I've found that structuring a Python project is one of the most often overlooked components of teaching the language. Worse, many developers get it wrong, stumbling through a jumble of common mistakes until they arrive at something that at least works. Here's the good news: you don't have to be one of them! In this installment of the Dead Simple Python series, we'll be exploring Setting Up The RepositoryBefore we delve into the actual project structure, let's address how this fits into our Version Control System [VCS]...starting with the fact you need a VCS! A few reasons are...
You've got plenty of options available to you. Git is the most obvious, especially if you don't know what else to use. You can host your Git repository for free on GitHub, GitLab, Bitbucket, or Gitote, among others. If you want something other than Git, there's dozens of other options, including Mercurial, Bazaar, Subversion (although if you use that last one, you'll probably be considered something of a dinosaur by your peers.) I'll be quietly assuming you're using Git for the rest of this guide, as that's what I use exclusively. Once you've created your repository and cloned a local copy to your computer, you can begin setting up your project. At minimum, you'll need to create the following:
That's right...our Python code files actually belong in a separate subdirectory! This is very important, as our repository's root directory is going to get mighty cluttered with build files, packaging scripts, virtual environments, and all manner of other things that aren't actually part of the source code. Just for the sake of example, we'll call our fictional project PEP 8 and NamingPython style is governed largely by a set of documents called Python Enhancement Proposals, abbreviated PEP. Not all PEPs are actually adopted, of course - that's why they're called "Proposals" - but some are. You can browse the master PEP index on the official Python website. This index is formally referred to as PEP 0. Right now, we're mainly concerned with PEP 8, first authored by the Python language creator Guido van Rossum back in 2001. It is the document which officially outlines the coding style all Python developers should generally follow. Keep it under your pillow! Learn it, follow it, encourage others to do the same. (Side Note: PEP 8 makes the point that there are always exceptions to style rules. It's a guide, not a mandate.) Right now, we're chiefly concerned with the section entitled "Package and Module Names"...
We'll get to what exactly modules and packages are in a moment, but for now, understand that modules are named by filenames, and packages are named by their directory name. In other words, filenames should be all lowercase, with underscores if that improves readability. Similarly, directory names should be all lowercase, without underscores if at all avoidable. To put that another way...
I know, I know, long-winded way to make a point, but at least I put a little PEP in your step. (Hello? Is this thing on?) Packages and ModulesThis is going to feel anticlimactic, but here are those promised definitions: Any Python ( Well...almost. There's one other thing you have to do to a directory to make it a package, and that's to stick a file called There is other cool stuff you can do with If you do forget So, if we look at our
project structure, Let's look at one a snapshot of my real-world projects,
Enter fullscreen mode Exit fullscreen mode (In case you're wondering, I used the UNIX program You'll see that I have one top-level package called I also have another special file in my top-level package: How import WorksIf you've written
any meaningful Python code before, you're almost certainly familiar with the
Enter fullscreen mode Exit fullscreen mode It is helpful to know that, when we import a module, we are actually running it. This means that any For example,
Naturally,
reading that, you might get a bit confused. I've had people ask me why the outer module (in this example, Of course, the above scenario is fictional: Import Dos and Don'tsThere are actually a number of ways of importing, but most of them should rarely, if ever be used. For all of the examples below, we'll imagine that we have a file called
Enter fullscreen mode Exit fullscreen mode Just for example, we will run the rest of the code in this section in the Python interactive shell, from the same directory as If we want to run the function
Enter fullscreen mode Exit fullscreen mode We would actually say that (By the way, don't confuse namespace with implicit namespace package. They're two different things.) The Zen of Python, also known as PEP 20, defines the philosophy behind the Python language. The last line has a statement that addresses this:
At a certain point, however, namespaces can become a pain, especially with nested packages. If we want to be able to use the
Enter fullscreen mode Exit fullscreen mode Note, however, that neither
Enter fullscreen mode Exit fullscreen mode In that terrible nested-package nightmare earlier, we can now say The Before long, though, you'll probably find yourself saying "But I have hundreds
of functions in my module, and I want to use them all!" This is the point at which many developers go off the rails, by doing this...
Enter fullscreen mode Exit fullscreen mode This is very, very bad! Simply put, it imports everything in the module directly, and that's a problem. Imagine the following code...
Enter fullscreen mode Exit fullscreen mode What do you suppose will happen? The answer is, Of course, since we usually don't know, or at least don't remember, every single function, class, and variable in every module that gets imported, we can easily wind up with a whole lot of messes. The Zen of Python addresses this scenario as well...
You should never have to guess where a function or variable is coming from. Somewhere in the file should be code that explicitly tells us where it comes from. The first two scenarios demonstrate that. I should also mention that the earlier
Some nesting of packages is okay, but when your project starts looking like an elaborate set of Matryoshka dolls, you've done something wrong. Organize your modules into packages, but keep it reasonably simple. Importing Within Your ProjectThat project file structure we created earlier is about to come in very handy. Recall my
Enter fullscreen mode Exit fullscreen mode In my Because I defined
Enter fullscreen mode Exit fullscreen mode This is called an absolute import. It starts at the top-level package, Some developers come to me with import statements more like It does, however, know about its parents. Because of this, Python has something called relative imports that lets us do the same thing like this instead...
Enter fullscreen mode Exit fullscreen mode The There's a lot of debate about whether to use absolute or relative imports. Personally, I prefer to use absolute imports whenever possible, because it makes the code a lot more readable. You can make up your own mind, however. The only important part is that the result is obvious - there should be no mystery where anything comes from. (Continued Reading: Real Python - Absolute vs Relative Imports in Python There is one other lurking gotcha here! In
Enter fullscreen mode Exit fullscreen mode Surely, since both these modules are in the same package, we should be able to just say Wrong! It will actually fail to locate However, we can use a relative import instead:
Enter fullscreen mode Exit fullscreen mode In that case, the single If you're familiar with the typical UNIX file system, this should start to make sense. However, keep in mind that those "levels" aren't just plain directories, here. They're packages. If you have two distinct packages in a plain directory that is NOT a package, you can't use relative imports to jump from one to another. You'll have to work with the Python search path for that, and that's beyond the scope of this guide. (See the docs at the end of this article.) __main__.pyRemember when I mentioned creating a Here's the contents of that file:
Enter fullscreen mode Exit fullscreen mode Yep, that's actually it! I'm importing my module Remember, I could also have said (Side Note: We could debate whether it's logical for me to have a separate The part that confuses most folks at first is the whole
When a module is run directly via Thus, You can see this in
action another way. If I added the following to the bottom of
Enter fullscreen mode Exit fullscreen mode ...I can then execute that module directly via Meanwhile, if I just run See how that works? Wrapping UpLet's review.
Of course, there are a lot more advanced concepts and tricks we can employ in structuring a Python project, but we won't be discussing that here. I highly recommend reading the docs:
Thank you to How do you import a structure in Python?Create and Install a Local Package. # setup.cfg [metadata] name = local_structure version = 0.1. 0 [options] packages = structure # setup.py import setuptools setuptools. ... . $ python -m pip install -e . This command will install the package to your system. ... . 7# Local imports 8from structure import files.. What are the 3 types of import statements in Python?There are generally three groups: standard library imports (Python's built-in modules) related third party imports (modules that are installed and do not belong to the current application) local application imports (modules that belong to the current application)
What is directory structure in Python?Your app is stored as a directory structure that you can clone using Git. Each Package, Module and Form consists of files and directories within this structure.
What is the best project structure for a Python application?Python doesn't have a distinction between /src , /lib , and /bin like Java or C has. Since a top-level /src directory is seen by some as meaningless, your top-level directory can be the top-level architecture of your application. I recommend putting all of this under the "name-of-my-product" directory.
|