Django suffers from a slow development process, due mainly to its commitment to backwards compatibility and API stability. This impacts the ability to introduce APIs, which must be developed in isolation and with limited user adoption until they land -- at which point, it becomes hard to make changes to them.
The main goal of this project is to implement the concept of "experimental APIs" in Django, as a mechanism to introduce changes to the framework that can be kept temporarily outside the usual deprecation/stability cycle.
It also proposes the concept of "third party contrib packages", as a mechanism for selected third party packages to integrate with django feeling as if they were part of the contrib API, without the need to incorporating them into the framework codebase, allowing thus a more agile development.
In addition, it is also proposed to test this new approach with the actual integration of an experimental feature, as well as some framework tooling supporting it.
There's already several ideas and discussions from the community regarding this topic, which can lay the basis for the work proposed in this document.
Most notably is the DEP2: experimental APIs, which already describes the concept of experimental APIs and provides some details around how they would fit within the current django release and versioning rules.
In parallel, there's a ticket in the new django feature proposals repository: Experimental feature flags and processes #3, where DEP2 is mentioned, and expands on the idea by proposing to implement feature flags as a mechanism to opt in or out of the experimental features. In the comments, other ideas are proposed as alternatives for this, such as an experimental module, or flags in the settings.
It is also established that there is a distinction to be made between changes/additions to existing core APIs, versus brand new APIs. For the latter, they have the advantage that they can be developed as a standalone third party package, and be used by the community before they become an official part of the framework itself (if they ever do). In this blog post, the author experiments with a formatting of third party packages that is compatible for seamless integration into django.contrib, and arguments the possibility of an officially maintained list of approved "third party contrib packages" in this format, end even be included in django releases. They would combine the advantages of core contrib packages , with the more agile development of third party packages.
Estimate: 2-3 weeks.
The first step would be to update and extend the existing DEP2, around experimental APIs, to cover:
- Exact definition of "experimental" APIs.
- Describe their lifecycle from idea phase, to implementation to release, taking into a account the newly updated contribution process starting from the new-features repo, through Trac and then in the code.
- Describe the rules they should adhere around the django release and versioning process , taking into account there's proposed changes around it. These would include rules around version phasing, as well as a requirement for tests and documentation and security maintenance.
- Actual implementation details: identify which pieces of documentation would need to be updated and how.
- Consider if these APIs should raise "experimental warnings" in the logs, in the same way that to-be-deprecated features do.
- Consider any implications around the new django-upgrade tool.
- Establish the distinction between "updates to core APIs" and "brand new APIs". Describe the idea of third party contrib packages as a possible alternative for the latter, possibly in its own dedicated DEP.
All of this work involves getting feedback and consensus from the community, so it will start by continuing the discussion in the existing ticket, in the DEP PR(s) when they exist, or by any other means.
Estimate: 1-2 weeks. Running total: 4-7 weeks.
After DEP2 is approved, any steps required for implementation should be carried. These will include mostly updating the Django docs, particularly around the API stability section, but possibly in other places too (such as the release process, or the new features process sections).
If it is decided that should come with logging warnings, these would need to be implemented into the django codebase, possibly in a similar fashion as deprecation warnings -- although actual implementation is TBD after discussion with the community.
Estimate: 2-3 weeks. Running total: 6-10 weeks.
Note: I keep referring to this concept with inconsistent naming, including "external contrib packages", "third party contrib packages", and using both "contrib packages" and "contrib APIs". I expect that during the process of drafting the idea into a DEP, an official name will emerge.
If (and only if) any DEP describing this process gets approval, its implementation should follow. This would mainly involve, again, the updating of any relevant docs, to be fully fleshed out in the DEP. Likely, some of the following might be impacted:
- The reference for contrib packages , to describe the existence and workings of the third party ones.
- The writing reusable apps tutorial, which could include a section on how to create third party contrib package. Alternatively, a dedicated howto guide might be more appropriate, since this is perhaps an advanced topic for the introductory tutorial (which could still link to the advanced howto).
As part of creating these howtos/tutorials, it may be considered creating an official template repository providing the scaffolding for contrib apps, to be referenced by the guides in the documentation.
Estimate: 2-8 weeks (4-8 weeks for feature flags). Running total: 8-18 weeks.
This would involve identifying a feature (or well-loved third party package) that is being considered for inclusion into the framework, and actually drive its release as the first official experimental feature, by using the new process.
This would serve as a way to promptly test the process and its implementation, and it may help identify if there any issues with it that we may have missed during the planning phases. If so, some corrections to the process may have to be implemented, likely in the form of updating docs.
Depending on the feature selected, especially if it comes in the form of a third party package, some substantial work might need to be done to adapt it to the right format. Which is why this section has such a wide range for the estimate.
Possible sources for features that are suitable for this are the django new-features kanban, or any official/curated lists of recommended third party packages, such as 1, 2 or 3. Approval and feedback from the community will be gathered in order to make the choice, so this will likely start in the form of a forum thread, or other means to drive discussion.
A personal proposal (subject to community approval) would be to actually implement a generic feature flags API as this "canary" experimental project. Seeing as this is a longstanding request by the community, it's a good candidate to become a first experimental feature, be it by integrating one of the popular third party packages (eg, waffle or django-flaqs), or just using them as an inspiration to write them in core from scratch. The reason for choosing the feature is that we can also leverage on it for the next milestone.
Estimate: 3-6 weeks. Running total: 11-24 weeks.
If time allows, as a way to support the now existing "experimental" APIs, implement a mechanism so that users can opt in or out of the experimental features more explicitly. Since experimental APIs are subject to change or even be removed altogether, it would be desiderable if they work in an opt-in manner, and be disabled by default, so that users that decide to adopt them do so knowingly.
As mentioned in the groundwork section, there already exists an official proposal for this in this issue. In there, specifically-purposed feature flags are suggested as such a mechanism, alongside simpler alternatives such as config settings.
Implementation will vary wildly depending on the solution chosen, as well as if core feature flags already exist in the framework or not (ie, if they were implemented in previous milestone), hence the stretched estimate.
The above estimates are made with the assumption of approximately 30 working hours per week. The GSoC assumes for a coding period of 12 weeks, extendable to 22.
I decided to plan on rough estimates, rather than a tightly scheduled timeline, since I think this is more realistic: a lot of the work is subject to consensus and decisions that haven't been fully fleshed out yet. Not just the timings, but the actual plans, might change direction along the way, which is why intentionally they are not described to the finest detail in this document: that's left as part of the implementation.
At the very least, I estimate that the core milestones should fit comfortably into the schedule of the program even on the higher estimates, while the extra milestones would be natural followups that can be included in the program if time allows (perhaps counting with the extension period), or can be worked on afterwards as just any regular open source contribution.
My name is Mar Bartolomé. I used to work as a software engineer doing Python web development, amassing around a decade of experience, although I left the industry a few years ago when I had children, to focus on caring for them during their infancy. I now trying to get reacquainted with tech by getting involved in open source, since I'v been a longstanding advocate.
I am especially fond of the Django community, which I found at the same time to be highly committed to quality but also very welcoming to new contributors. I got introduced to it last year, when I participated in the Djangonauts program as a mentee, contributing documentation and bugfixes to DjangoCMS, and as a team captain in the currently running session. I'd like to become further involved in the community, and become a regular contributor. This project looks like a great opportunity to get immersed not only in the django core codebase, but also in its development lifecycle and governing processes.
For my full collection of social profiles, see: https://mar.coconauts.net/