Guido, Some Guys, and a Mailing List: How Python is Developed
by Brett Cannon (brett at python.org)
Introduction
Software does not make itself. Code does not spontaneously come from the ether of the universe. Python is no exception to this rule. Since Python made its public debut back in 1991 many people beyond the BDFL (Benevolent Dictator For Life, Guido van Rossum) have helped contribute time and energy to making Python what it is today; a powerful, simple programming language available to all.
But it has not been a random process of people doing whatever they wanted to Python. Over the years a process to the development of Python has emerged by the group that heads Python's growth and maintenance; python-dev. This document is an attempt to write this process down in hopes of lowering barriers preventing people from contributing to the development of Python.
Tools Used
To help facilitate the development of Python, certain tools are used. Beyond the obvious ones such as a text editor and email client, two tools are pervasive in the development process.
Roundup is used by python-dev to keep track of feature requests, reported bugs, and contributed patches. A detailed explanation on how to use Roundup is covered later in General Roundup Guidelines.
Subversion (also called svn) is a networked file versioning system that stores all the files that make up Python. SVN allows the developers to have a single repository for the files along with being able to keep track of any and all changes to every file. The basic commands for using SVN can be found in the developer FAQ, along with a multitude of tutorials spread across the web. The Subversion repository is accessible at http://svn.python.org/view/.
Communicating
Python development is not just programming. It requires a great deal of communication between people. This communication is not just between the members of python-dev; communication within the greater Python community also helps with development. Several mailing lists and newsgroups are used to help organize all of these discussions.
In terms of Python development, the primary location for communication is the python-dev mailing list. This is where the members of python-dev hash out ideas and iron out issues. It is an open list; anyone can subscribe to the mailing list. While the discussion can get quite technical, it is not all out of the reach for even a novice and thus should not discourage anyone from joining the list. Please realize, though, this list is meant for the discussion of the development of Python; all other questions (including those about development in Python) should be directed somewhere else, such as python-list. Along with this, a level of etiquette is expected to be maintained. A lack of manners will not be tolerated.
When the greater Python community is involved in a discussion, it always ends up on python-list. This mailing list is a gateway to the newsgroup comp.lang.python. This is also a good place to go when you have a question about Python that does not pertain to the actual development of the language.
Using svn allows the development team to know who made a change to a file and when they made their change. But unless one wants to continuously update their local checkout of the repository, the best way to stay on top of changes to the repository is to subscribe to Python-checkins. This list sends out an email for each and every change to a file in Python. This list can generate a large amount of traffic since even changing a typo in some text will trigger an email to be sent out. But if you wish to be kept abreast of all changes to Python then this is a good way to do so.
The Python-bugs-list sends out an email for all changes to bug report or patch items in Roundup. This list, just like Python-checkins, can generate a large amount of email traffic. It is in general useful to people who wish to help out with the development of Python by knowing about all new submitted patches as well as any new developments on preexisting ones.
The New issues list restricts such email messages to just new issues.
The Actual Development
Developing Python is not all just conversations about neat new language features (although those neat conversations do come up and there is a process to it). Developing Python also involves maintaining it by eliminating discovered bugs, adding and changing features, and various other jobs that are not necessarily glamorous but are just as important to the language as anything else.
General Roundup Guidelines
Since a good amount of Python development involves using Roundup, it is important to follow some guidelines when handling a tracker item (bug, patch, etc.). Probably one of the most important things you can do is make sure to set the various options in a new tracker item properly. The submitter should make sure that the Type, Components, and Versions are all set to reasonable values. The remaining values (Assigned To, Status, and Resolution) should in general be left to Python developers to set. The exception to this rule is when you want to retract a patch; then "close" the patch by setting Status to "closed" and Resolution to whatever is appropriate.
XXX closing is currently not possible that way.
Make sure you do a cursory check to make sure what ever you are submitting was not previously submitted by someone else. Duplication just uses up valuable time. If you must create a new item (for instance, you are posting a patch for a bug report), please place cross-references in both tracker items.
And please do not post feature requests, bug reports, or patches to the python-dev mailing list. If you do you will be instructed to create an appropriate Roundup tracker item. When in doubt as to whether you should bring something to python-dev's attention, you can always ask on comp.lang.python; Python developers actively participate there and move the conversation over if it is deemed reasonable.
Feature Requests
A feature request (type rfe) means you have no plans on actually implementing your idea by writing a patch. On occasion people do go through the features requests to see if there is anything there that they think should be implemented and actually do the implementation. But in general do not expect something put here to be implemented without some participation on your part.
The best way to get something implemented is to campaign for it in the greater Python community. comp.lang.python is the best place to accomplish this. Post to the newsgroup with your idea and see if you can either get support or convince someone to implement it. It might even end up being added to PEP 42 so that the idea does not get lost in the noise as time passes.
Bug Reports
Think you found a bug? Then submit a bug report with Roundup. Make sure you clearly specify what version of Python you are using, what OS, and under what conditions the bug was triggered. The more information you can give the faster the bug can be fixed since time will not be wasted requesting more information from you.
Patches
Create a patch tracker item with Roundup for any code you think should be applied to the Python svn tree. For practically any change to Python's functionality the documentation and testing suite will need to be changed as well. Doing this in the first place speeds things up considerably.
Please make sure your patch is against the svn repository. If you don't know how to use it (basics are covered in the developer FAQ), then make sure you specify what version of Python you made your patch against.
In terms of coding standards, PEP 8 specifies for Python while PEP 7 specifies for C. Always try to maximize your code reuse; it makes maintenance much easier.
For C code make sure to limit yourself to ANSI C code as much as possible. If you must use non-ANSI C code then see if what you need is checked for by looking in pyconfig.h . You can also look in Include/pyport.h for more helpful C code. If what you need is still not there but it is in general available, then add a check in configure.in for it (don't forget to run autoreconf to make the changes to take effect). And if that still doesn't fit your needs then code up a solution yourself. The reason for all of this is to limit the dependence on external code that might not be available for all OSs that Python runs on.
Be aware of intellectual property when handling patches. Any code with no copyright will fall under the copyright of the Python Software Foundation. If you have no qualms with that, wonderful; this is the best solution for Python. But if you feel the need to include a copyright then make sure that it is compatible with copyright used on Python (i.e., BSD-style). The best solution, though, is to sign the copyright over to the Python Software Foundation.
When you think you're ready, review your own patch (see the instructions in Helping Out). Then try to review a few other patches as well; many patches are never applied because no one has the time to review them. If you review enough patches from other people, someone with checkin privileges is more likely to spend some time on your patch.
Changing the Language
You understand how to file a patch. You think you have a great idea on how Python should change. You are ready to write code for your change. Great, but you need to realize that certain things must be done for a change to be accepted. Changes fall into two categories; changes to the standard library (referred to as the "stdlib") and changes to the language proper.
Changes to the stdlib
Changes to the stdlib can consist of adding functionality or changing existing functionality.
Adding minor functionality (such as a new function or method) requires convincing a member of python-dev that the addition of code caused by implementing the feature is worth it. A big addition such as a module tends to require more support than just a single member of python-dev. As always, getting community support for your addition is a good idea.
With all additions, make sure to write up documentation for your new functionality. Also make sure that proper tests are added to the testing suite.
If you want to add a module, be prepared to be called upon for any bug fixes or feature requests for that module. Getting a module added to the stdlib makes you by default its maintainer. If you can't take that level of responsibility and commitment and cannot get someone else to take it on for you then your battle will be very difficult; when there is not a specific maintainer of code python-dev takes responsibility and thus your code must be useful to them or else they will reject the module. There is also the possibility of having to write a PEP (read about PEPs in Changing the Language Proper).
Guido has also stated that for any new module it has to have been accepted and widely used by the greater Python community before it can be considered for addition to the stdlib. This is to prevent bloat in the stdlib by making sure that any new modules will actually be used and not just by the person who wrote the module and a few people who supported its addition.
Changing existing functionality can be difficult to do if it breaks backwards-compatibility. If your code will break existing code, you must provide a legitimate reason on why making the code act in a non- compatible way is better than the status quo. This requires python- dev as a whole to agree to the change.
Changing the Language Proper
Changing Python the language is taken very seriously. Python is often heralded for its simplicity and cleanliness. Any additions to the language must continue this tradition and view. Thus any changes must go through a long process.
First, you must write a PEP (Python Enhancement Proposal). This is basically just a document that explains what you want, why you want it, what could be bad about the change, and how you plan on implementing the change. It is best to get feedback on PEPs on comp.lang.python and from python-dev. Once you feel the document is ready you can request a PEP number and to have it added to the official list of PEPs in PEP 0.
Once you have a PEP, you must then convince python-dev and the BDFL that your change is worth it. Be expected to be bombarded with questions and counter-arguments. It can drag on for over a month, easy. If you are not up for that level of discussion then do not bother with trying to get your change in. If you manage to convince a majority of python-dev and the BDFL (or most of python-dev; that can lead to the BDFL changing his mind) then your change can be applied.
As with all new code make sure you also have appropriate documentation patches along with tests for the new functionality.
Changes to the documentation
The Python documentation is maintained using reStructuredText with many additional features specific for use in documenting Python.
If you're working on an issue in the Roundup trackers that contains non-reST documentation, and you don't feel you can convert the text to reST yourself, you can assign the issue to one of the reStructuredText wranglers as well.
Helping Out
Many people say they wish they could help out with the development of Python but feel they are not up to writing code. There are plenty of things one can do, though, that does not necessarily require you to write code. Regardless of your coding abilities, there is something for everyone to help with. Below is an attempt to discuss all possible options one has in helping out with development.
For feature requests, adding a comment about what you think is helpful. State whether or not you would like to see the feature. You can also volunteer to write the code to implement the feature if you feel up to it.
For bugs, stating whether or not you can reproduce the bug yourself can be extremely helpful. (Do give details about your system; if the bug shows up only on Linux, or only on Windows XP, that makes it easier to track down.) If you can write a fix for the bug that is very helpful as well; start a patch item and reference it in a comment in the bug item.
There is always a need for more people to test and review patches! Apply the patch and run the testing suite. You can do a code review on the patch to make sure that it is good, clean code. If the patch adds a new feature, comment on whether you think it is worth adding. If it changes functionality then comment on whether you think it might break code; if it does, say whether you think it is worth the cost of breaking existing code. Help add to the patch if it is missing documentation patches or needed regression tests. (Please check for existing tests both in the code itself, and in the test package.)
For language changes, make your voice be heard. Comment about any PEPs on comp.lang.python so that the general opinion of the community can be assessed.
If there is nothing specific you find you want to work on but still feel like contributing nonetheless, there are several things you can do. The documentation can always use fleshing out. Adding more tests to the testing suite is always useful. Running the test suite itself (see the documentation for the 'test' package in the standard library on how) and reporting any failed test is always a great help. Contribute to discussions on python-dev, comp.lang.python, or one of the SIGs (Special Interest Groups). Just helping out in the community by spreading the word about Python or helping someone with a question is helpful.
If you really want to get knee-deep in all of this, join python-dev. Once you have been actively participating for a while and are generally known on python-dev you can request to have checkin rights on the svn tree. It is a great way to learn how to work in a large, distributed group along with how to write great code.
And if all else fails give money; the Python Software Foundation is a non-profit organization that accepts donations that are tax- deductible in the United States. The funds are used for various thing such as lawyers for handling the intellectual property of Python to funding PyCon. But the PSF could do a lot more if they had the funds. One goal is to have enough money to fund having Guido work on Python for a full year full-time; this would bring about Python 3. Every dollar does help, so please contribute if you can.
Conclusion
If you get any message from this document, it should be that anyone can help with the development of Python. All help is greatly appreciated and keeps the language the wonderful piece of software that it is.
