While the build tool Pants has supported building and testing Python 3 code for a long time, until today, Pants itself was built entirely with Python 2. As of today, the tool can now be run entirely through Python 3. In about two months, all new versions of Pants will require Python 3 to run the tool.
This post walks through how during my summer internship at Foursquare I led this project to migrate Pants to Python 3, the major roadblock to our own internal migration.
[Important clarification: Pants will still be able to support users’ Python 2 codebase, it just will need a Python 3 interpreter discoverable to run the tool.]
Wait, what is Pants?
Pants is a build system to allow working with enormous monorepos that may use multiple languages—including Python, Java, and Scala—dozens of tools, and thousands of dependencies. It’s a collaborative open source project, built and used by Twitter, Foursquare, Medium, Square, and other companies and independent developers.
Here at Foursquare, every engineer’s entrypoint to our codebase begins with Pants, used to execute commands like ./pants test and ./pants run.
Migrations need a kickstarter
Migrations are grindy. While most organizations know they have only nine months to deal with Python 2 until losing official support, that does not make the process of deciphering and modernizing thousands of lines of code you did not write any less time-consuming or repetitive.
For both Foursquare and Pants, we had wanted to see this migration happen, but in neither case wanted to prioritize the daunting task. Further, Foursquare itself had yet to seriously begin its own migration.
Originally for my summer internship, I was on a completely different team than our Developer Systems team, which helps maintain Pants. When I found out I would have to write my project in Python 2 and couldn’t use type hints, I started poking around with how I could get us to something more modern. I presented a plan at our weekly developer meeting for how we could feasibly get both type hints and migrate to Python 3 for Foursquare’s internal codebase, and created a proof of concept. This led to an offer to change teams to lead our migration effort with their support. Two weeks in, we realized Pants would be the major roadblock to Foursquare’s migration, so my manager offered me the opportunity to spearhead Pants’ migration. (Facebook’s leader of their Python 3 migration had a similar experience).
The takeaway is that you do not have to already be on the internal infrastructure team to make your organization’s migration happen. Likewise, spearheading a project is never intended to be supported from a sole contributor. A successful migration is impossible without the support of a strong team.
Incremental migration that’s Python 3 first
Joel Spolsky claimed in 2000 that The Big Rewrite is the “single worst strategic mistake” a software project can make.
Rewriting Pants to Python 3 in one fell swoop would not work. Pants is simply too big of a project (150,000 lines of code) with too many unrelated changes happening throughout the migration. Even worse would be maintaining Python 2 and Python 3 branches, as many libraries attempted when Python 3 first came out.
Instead, we took the approach recommended by the Django authors to write Python 3 code that’s compatible with Python 2, rather than the opposite of trying to get our Python 2 code to work with Python 3. For any gain we made in compatibility, we added continuous integration (CI) to ensure that we did not ever go backwards in the migration.
To allow us to pursue this incremental approach, we implemented several CI blacklists. Anything not on the blacklist would default to Python 3, and anything on the list would run with Python 2. Every night, we would run a cron job entirely with Python 2 to check for Python 2 regressions. Through this approach, we were able to first get all unit tests passing with Python 3, then integration tests running as a spawned Python 3 subprocess with Python 2 still for the main process, and finally, all tests using full Python 3.
7 out of 20 of our integration test shards running blacklisted tests with Python 2.
Beyond CI blacklists, the lynchpin of this incremental approach migration rested on the python-future library— essentially a souped up version of the popular six library—which allows you to write Python 3 code that also works with Python 2. For anyone contemplating a migration to Python 3, we unequivocally recommend using the python-future library as the center of your migration.
Automate the boring stuff (carefully)
Python 3 migrations are tricky in that most of the issues boil down to the same 2-3 tedious problems, but they involve semantic changes that require human input to address.
Rather than trying to automate everything, what works best is automation with human input.
For example, the first portion of the project involved calling the python-future library’s futurize –stage2 script over the whole codebase, –stage2 meaning it would make unsafe semantic changes to the codebase. For us to port any file, we would have to:
- Call futurize –stage2.
- Review the changes and revert any over-conservative changes.
- Add the python-future library to our dependencies list in that file’s BUILD entry.
- Run our autoformatter.
- Find and then run the corresponding tests to check for regressions.
While most of this could be automated, step 2 could not be. The Futurize script could not understand the original semantics, so it eagerly applied changes often not necessary that decreased performance and readability. We had to decipher what the original author intended and clean it all up.
So, we wrote a simple script futurize.py that would automatically do every step, but pause at step 2 for human input.
The first change is necessary, but the second is unnecessary and hurts performance.
Thanks to this script, we were able to recruit two other engineers—who had never worked closely with Pants or Python migrations before—to spend a week running this script over the whole codebase without making any significant errors.
Through this, we learned that it’s important to automate as much of your migration as possible, or you will want to give up from the tedium; but be mindful to build human review into the script itself.
All the scripts used to automate this project can be found and used here
Rust and FFI interop
At the same time we led this Python 3 migration, a team at Twitter had been rewriting Pants’ underlying engine to use Rust. So, a substantial portion of the codebase required interop with Rust through the CFFI library. Further, we support interop with C and C++ code, including through the Python C API.
With Python migrations, FFI (foreign function interface) is one of the less traveled and documented pain points. After many confusing panic! stack traces, the main takeaways we learned from porting FFI code boiled down to:
- Be explicit with how you serialize your text data, both from Python and to Python. Sending unicode when you meant to send bytes can result in difficult-to-debug errors.
- Compile Rust code with the interpreter you are running it with. Our Rust Native Engine builds against Python.h, which varies subtly between Python 2 and Python 3. We realized we must build two separate engine binaries and use the appropriate binary for the interpreter we are currently using.
- For porting C and C++ code, follow this great guide.
Looking forward
In June, when Pants 1.17.0 is released, we will require Python 3.6+ to run Pants.
Requiring Python 3 brings a bright future for the Pants project, allowing us to:
- Add type hints for fewer bugs and more readable code. (Possible in Python 2, but not as ergonomic).
- Use the asyncio library to simplify and expand our concurrency support.
- Never again have unicode bugs (or at least we can dream).
- Use f-strings for better readability and performance.
- Use keyword-only arguments to enforce higher quality code.
- Use enums and dataclasses to improve our data models.
- Port our bash scripts to Python thanks to subprocess.run().
While to the end user, no one will see what Python version we use, this migration represents a technical investment that makes for happier and more productive engineers internally. In turn, it improves our systems and technology to help other businesses better understand how people move through the real world.
