code

How companies should adapt interviews to AI

Interviewing in the age of AI


_Not a single sentence of this article has been written by AI._

Most management questions can be answered by "it depends" and "I know it when I see it". A couple of weeks ago a peer asked me "how do you know when to step in and give directions as a manager?" My first hunch was to reply "based on my experience (and many mistakes)".

We had a good back and forth on this question, so I thought this would make a good blog post. Here it is.

## Introduction

Managers are here to... manage. One would be right to question the value of managers **who do not make any decision** or never step in to help their team. Some organizations do confine managers to pure people management activities (hiring, performance management, administrative tasks such as PTO approval), but those are rarely the best-in-class, most effective organizations.

The other extreme is not any better. A manager who's always intervening and making every single decision will soon be left on their own. **No knowledge worker wants to be confined to pure execution**. The best employees want agency and autonomy. Micro-management is the surest way to create attrition within your team.

One of my favorite management books of all time is [Turn the Ship Around!](https://davidmarquet.com/books/turn-the-ship-around-book/) by former Navy captain David Marquet. Most management books are repetitive and boring to read - this is not one of them. Its main thesis might seem common-sense nowadays: replacing the leader-follower model with a leader-leader model leads to a more effective team. It also offers many useful techniques to empower your team. If you're looking for a more systematic treatment of this topic, I highly encourage you to read it.

**This is a very nuanced topic**, I'll provide you with a few ideas and models that can help you think about whether and how to step in.

## Having to step in is a form of management failure

**None of this article should be construed as a one-sided justification for manager's intervention**.

One of my previous managers told me he would postmortem every time he had to step in to make a call for his team. That's a pretty good idea:

- What **competency** do we lack as a team? Should we grow it internally (coaching, training, documentation, etc.)? Externally (hiring)?
- What piece of **context** was my team lacking?
- Was the "**why**" behind this project clear?

You need to be able to trust your team to **scale your impact**.

## The what: "below the waterline" mistakes

There is a corollary to providing your reports with agency: **you must let them make mistakes**, which is - as all parents know - the best way to grow and build competency.

We can make a first distinction. A ship's hull can be divided in two parts. Any hole "**below the waterline**" will surely lead to the ship's demise. "**Above the waterline**" holes are not as impactful.

So, when considering a team decision you disagree with, ask yourself: "if it's a mistake, will lead to a hole 'below the waterline'?". If so, it's a good sign you should intervene to save your team/employee from a fatal, potentially career-limiting mistake. The same goes with **one-way decisions** that are very costly to reverse.

## The what: disagree & commit to difference in taste

You also need to distinguish between **potential mistakes** and **stylistic details**. Most decisions involve a nuanced understanding of tradeoff, for which taste matters (especially when it comes to system design). For instance, when reviewing code, you should approve liberally if your only feedback is about style[^1].

If you have a strong opinion about how things should be done (most managers do), then it's worth at least spending the time with the individual contributor in charge of the project to pitch your idea. You're only **trying to convince them** while acknowledging that they're in charge of the final decision.

Depending on the importance of this decision, and the tradeoff horizon involved, it might be a good idea to capture how you're thinking about the same problem **in writing**, so that you can reference it later. The goal is not to be able to say "I told you so!", but to ensure the team as a whole extract all the learnings from that past decision.

## The what: the vision is usually a single person's responsibility

The strongest visions are the most consistent ones. Consistency almost definitionally requires a **single origin**. The manager is usually uniquely positioned to provide that vision, because they have the most business context, and the most credibility given their differentiated role.

Note that **it does not follow that this vision solely _comes_ from the manager**. Quite the contrary. Sometimes, the team is relatively aligned on a technical topic (e.g., "we need better observability", or "we have too many micro-services", or "we need to invest in reliability") and all it needs is a manager who reinforces that vision with some key design ideas.

## "When" to step in: seizing the moment

Choosing the right moment to intervene as a manager requires a lot of judgment. Here are a few guardrails:

- **Lots of ambiguity**: good managers step in to help their team escape analysis-paralysis by providing design constraints. At one point in my career I made the mistake to let the team design a completely new service with _carte blanche_. They quickly got lost and spent two weeks challenging every single part of the infrastructure we had built so far. Once I stepped in to provide some constraints (a soft deadline, and coached them to reuse the existing stack, focusing on the differentiating features of this new service), the situation unblocked itself.
- **Design conflict**: a good manager won't let a design disagreement block a project's progress for too long. That's another mistake I've made: an engineer was driving a new system project, and would change their mind based on their interlocutor (they wanted to follow approach A, a senior tech lead convinced them to do solution B, and during RFC review another team's senior IC drove them towards solution C). While all three solutions had different tradeoffs, I let the situation linger for too long before stepping in to clarify that this was their project, and that their solution A was a perfectly reasonable one if they believed in it.

## "With whom": the TRM (task-relevant maturity)

Andy Grove coined the TRM (**task-relevant maturity**) acronym in _High Output Management_: your amount of involvement will range from full delegation (high TRM) to close direction (low TRM).

While it usually correlates with the individual's seniority, it's a good idea to challenge that bias. For instance, some very senior folks have **much less experience** with public speaking than others, and in that case, it makes sense for you to review the presentation material, ask for a rehearsal, etc. beforehand.

I find this concept to be useful for a-posteriori reviews too: when you review your level of engagement, you should verify that it was appropriate for the individual's particular TRM at the time.

## The "how": different ways to intervene

There are many ways for a manager to intervene and picking the right way evidently depends on the situation:

- **Providing a constraint or a pattern**: for instance, all new features must be behind feature flags.
- **Changing the project staffing**
- **Killing or redirecting a project**
- **Influencing direction** by encouraging a direct report to double down on their chosen path, or re-stating a clear vision, providing a target metric.
- **Making a direct decision**, overriding past directions. (should be postmortem'd)

## A principled "how"

A good manager intervention will ensure that the team **learns** so that next time, it can be self-sufficient:

- **Principled**: if you need to step in, you'll have to explain why you're doing so and what principles are helping you make a decision that your reports weren't able or willing to make. For instance, you should be able to pinpoint the missing aspects of a tradeoff analysis.
- **Contextualized**: perhaps your report were missing a critical piece of context that you're providing (in which case, it's a good idea to reflect on why you/it wasn't clear in the first place).
- **Empowering**: it's not always trivial to do, but a good manager intervention strikes a good balance between "unblocking clarity" & "leaving the employee at the helm".

## Further resources

This is such a deep topic. I could only provide some high-level thoughts. Here are some resources to go further (as always, check out my [engineering-management](https://github.com/charlax/engineering-management) repository for more.

About Decisions:

- [Reversible and Irreversible Decisions](https://fs.blog/reversible-irreversible-decisions/)
- [Disagree and Commit](https://tomtunguz.com/disagree-and-commit/)
- [Ask Questions, Repeat The Hard Parts, and Listen – Rands in Repose](https://randsinrepose.com/archives/ask-questions-repeat-the-hard-parts-and-listen/)

Other articles:

- [Unexpected Anti-Patterns for Engineering Leaders](https://review.firstround.com/unexpected-anti-patterns-for-engineering-leaders-lessons-from-stripe-uber-carta/), Will Larson
- [Balancing Engineering Cultures: Debate Everything vs. Just Tell Me What To Build](https://www.fishmanafnewsletter.com/p/balancing-engineering-cultures-debate-vs-do)
- [The Counterintuitive Art of Leading by Letting Go](https://www.behance.net/blog/the-counter-intuitive-art-of-leading-by-letting-go)
- [Being in the details](https://www.theengineeringmanager.com/managing-managers/being-in-the-details/)
- [An Engineering Team where Everyone is a Leader](https://blog.pragmaticengineer.com/a-team-where-everyone-is-a-leader/), Gergely Orosz
- [Developing leadership styles](https://lethain.com/developing-leadership-styles/), Will Larson
- 📖 [Leading Snowflakes: The New Engineering Manager's Handbook](https://leadingsnowflakes.com/), Oren Ellenbogen
- 📖 [Turn the Ship Around!](https://davidmarquet.com/books/turn-the-ship-around-book/), David Marquet

[^1]: For instance, if you're passionate about Functional Programming, you probably don't need to ask your reviewee to rewrite all their code to follow those patterns.


When should a manager step in?


Update (Mar 15, 2026): there is a good conversation on [Hacker News](https://news.ycombinator.com/item?id=47380231). The main contention seems to be about whether AI really change the game. For companies which already had their docs in their repos - not really (full disclosure: I've always been in favor of this approach - and I agree that AI agents reinforce the need for following best practices). For those who were not yet following this approach, I'm arguing that _now_ is a good time to do so. I agree that the same thing that helps AI, helps humans.

> "The palest ink is more reliable than the most powerful memory."
>
> – Chinese proverb

AI changes the game when it comes to having all your docs in your repository: it's never been that easy to keep them up to date!

I've always been a fan of having documentation living alongside the code:

- **Version control**: just like code, documentation evolve. Why use a different versioning control system when you're already using git? Especially when multiple people are changing docs at the same time, with potentially conflicting changes.
- **Proximity with code**: e.g., `rg` or `grep` will yield code and documentation results, making it much easier to keep it up to date.
- **Formal review and approval**: in the spirit of [documentation-driven development](https://gist.github.com/zsup/9434452), starting with a review of documentation updates help understand the final product/API. (For active collaboration other tools, e.g., Google Docs still provide a superior UX). This is also a much simpler and explicit workflow for enabling collaboration: if you spot a mistake in another team's doc, you can send a PR (instead of a lengthy, "ask first", "make edits", etc.)
- **Automatic generation**: when using a different system for hosting the docs (Google Docs, Confluence, Notion, etc.), it's quite laborious to copy-paste APIs and example code. There are many tools (e.g., Sphinx's [autodoc](https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#ext-autodoc), [jsdoc](https://jsdoc.app/), javadoc, docusaurus) that can generate API docs directly from the code.
- **Testing**: static code examples in documentation are a good start, but it's even better when they're tested, which you can do when running code examples in docs is part of your continuous integration process. See Python's [doctest](https://docs.python.org/3/library/doctest.html) for example. In a way, the documentation is the spec.
- **Efficient editing**: you benefit from all your text editor tools, and can script mass-changes.

## We will be spending more time writing docs

First observation: AI agents have considerably increased the proportion of markdown[^1] files in commits. That's usually because folks check out the agent's implementation, which [is a very good idea](/posts/2026-01-08-write-and-checkout-the-plan). It's also because you save a lot agentic iteration time when you write rules files (`.mdc` files) to guide agents' execution. So, whether or not you agree with the thesis, it is happening.

I would submit that 80% of the rules file could have been documentation instead, or potentially are already documented elsewhere. **Just like code should be primarily written for humans to read, all files in a repository is written primarily for humans to review**. This also applies to rules files, skills, etc. created to guide agentic execution ([skills are docs](https://passo.uno/skills-are-docs/). Rules files are reading more and more like style guides and best practices that we never bothered writing but probably should have solidified.

The frontier between AI-only markdown and human-only is so blurry that I could see rules files completely disappearing and being replaced by documentation.

This is also consistent with **engineers shifting their focus left**. Engineering tooling has trended towards higher and higher abstractions, from machine code to C, to dynamic languages, to SDK, and now to not even writing code and only focusing on the spec, and guidelines. Just like we don't review the machine code produced by the compiler, there might be a day where we don't review the code generated by an LLM provided it respects the [harness](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html), the specs and the guidelines (security will be a key concern there). In that world, we'll spend most of our energy reviewing the specs, the harness, the guidelines. Conclusion: those docs need to be written first and foremost for humans review.

## Why AI makes it even more meaningful to move docs into your repo

**AI agents solve stale docs**. A common objection to writing docs is: "why bother? read the code - the code is always up to date". (the same line of arguments would apply to brushing your teeth). AI agents solve this problem. They take away the laborious work to ensure code & documentation alignment (either in PR, or with specialized review agents that look for documentation inconsistencies). Quite game-changing.

**AI agents benefit from higher level context**. Moving your docs (including, perhaps, your architecture proposals - RFCs, your product specs - PRDs, etc.) will provide that extra context.

**Materialized plans with findings will save token & iteration time**. Imagine researching "the best way to do X" in a massive codebase. You will spend a lot of tokens finding the answer to that question. Documenting the answer to that question and [materializing it in the repository](/posts/2026-01-08-write-and-checkout-the-plan) will enable your colleague to skip that research step later (and keep it up-to-date with extra learnings, best practices, etc.!). This is especially the case for things that agents can't infer from the code, or are difficult to infer: typically, infra-related things you learned by deploying your code to production. For instance, I spent about two weeks researching and iterating on structured logging best practices - I materialized that into a "metaplan" that other teams can use directly, saving everyone (including agents!) a ton of time!

## Answer to objections

**Objection: you could use MCP and other approaches (skills) to give agent access to your documentation**. Answer: the same arguments I laid out in the beginning still apply, especially the version control piece. Most documentation system are not designed for fast iteration with strong concurrency control.

**Objection: waiting for a code review for docs will deter from updating docs**. Answer: (1) What if you weren't the one writing those docs? (AI can do it in the same PR) (2) Who says all repo content change need to go through review? (3) As we shift more and more left, won't the documentation change or implementation plan be the most important thing to review? (4) This is a better process for enabling doc contribution from outside your team.

**Objection: AI agents write long, convoluted docs**. Answer: well, most humans do as well :). Just like code, you should (1) review the agent's work, (2) fix the agent's work, and (3) write your own docs (like this article: none of this content has been generated by AI!). Putting it into version control makes it MUCH easier and safer (reviews! history!) to iterate.

**Objection: do I really need to move all my documentation?** Answer: I'd say yes, at this point. Not the fleeting docs, but everything that provide useful context about the codebase, including RFCs.

**Objection: [your_preferred_tool] is better at [tables/schemas/links]**. Answer: AI is getting incredibly good at those too.

**Objection: [your_preferred_tool]'s UX is better for human collaboration**. Answer: yes, Google Docs is still much better for active collaboration, so it's fair to continue using it for that use case. But once the documentation is in a good place, I would move it in the repo (Google Docs has a useful "Copy as Markdown" feature that I use all the time).

**Objection: non-engineers usually don't have repo access**. Answer: (1) You can deploy your docs on an internal-only website. (2) There is clear [trend with non-engineer code access](https://justinjackson.ca/claude-code-ruin/) (which poses some interesting security challenges).

**Objection: images will bloat your repo**. Answer: I've never been a fan of static images in docs: they're not grep-able and difficult to keep to up-to-date. Luckily, most markdown flavors support diagram format such as [mermaid](https://mermaid.js.org/) (also supported on Github). Not only will agents benefit from diagram (for instance, documenting the request flow for some of your core features will be a huge token-saver: AI tend to spend a lot of cycles rebuilding that context), but it will also be much easier to keep up-to-date.

## References and articles

As always, there are more resources on my repo [charlax/professional-programming](https://github.com/charlax/professional-programming)

About AI and docs:

- [Specifications Are Becoming the Real Source Code](https://7ynk3r.github.io/2026/01/19/specifications-are-becoming-the-real-source-code/)
- [Getting AI to Work in Complex Codebases](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md)
- [Skills are docs, and docs need tech writers](https://passo.uno/skills-are-docs/)

About documentation in general:

- [Documentation-Driven Development](https://gist.github.com/zsup/9434452)
- [Writing automated tests for your documentation](https://krausefx.com/blog/writing-automated-tests-for-your-documentation)
- [Documentation is king](https://speakerdeck.com/kennethreitz/documentation-is-king), Kenneth Reitz
- [The documentation system](https://documentation.divio.com/)
- [Write documentation first. Then build.](https://reproof.app/blog/document-first-then-build)
- [Diátaxis](https://diataxis.fr/): a systematic approach to technical documentation authoring: tutorials, how-to guides, technical reference, and explanation.

Paid software:

- https://www.mintlify.com/
- https://www.gitbook.com/ (has an open source front-end: https://github.com/GitbookIO/gitbook)

[^1]: You don't [have to](https://mister-gold.pro/posts/do-not-use-markdown-for-docs/) use Markdown. [AsciiDoc](https://docs.asciidoctor.org/asciidoc/latest/syntax-quick-reference/), [Markdoc](https://markdoc.dev/docs/syntax) and [reStructuredText](https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html) provide more powerful and structured alternatives.


AI agents address the remaining concerns with having your documentation alongside your code in version control

It's time to move your docs in the repo


Each AI-aided coding session should start in planning mode. It's become a pretty commonplace recommendation (Boris Cherny, creator of Claude Code, [makes this point here](https://www.reddit.com/r/ClaudeAI/comments/1q2c0ne/claude_code_creator_boris_shares_his_setup_with/)). I also recommend checking that plan into the repository.

## Summary

1. Start most coding sessions in planning mode. Iterate on it until it's in a good place.
2. Get the plan reviewed and approved. Merge it into the repo.

## Why planning makes sense

Separate your process into two PRs:

1. The PR with the plan.
2. The PR with the implementation.

The main reason is that it mimics the classical research-design-implement loop.

The first part (the plan) is the RFC. Your reviewers know where they can focus their attention at this stage: the architecture, the technical choices, and naturally their tradeoffs. "It's easier to use an eraser on the drawing board, than a sledgehammer at the construction site"

There are many other reasons why this approach make sense:

- It's much easier to "checklist" that you covered everything at this stage: monitoring, alerting, analytics, testings, docs, runbook, etc.
- The AI's context can be focused on the technical decisions without being wasted on the implementation details minutia.

## Why checking the plan into the repo makes sense

By checking the plan into the (git) repository, you achieve many things at the same time:

- The plan archives the decisions made, their rationale, and tradeoff (similarly to [ADR - Architecture Decisions Records](https://adr.github.io/)), which is useful when you need to understand the architecture's history.
- It documents the implementation (note: AI agents are great at keeping docs up to date), which is useful for debugging or runbooks.
- It might serve as a template for future similar projects.
- It might serve as inspiration for other teams looking at similar issues.
- It will be found and mentioned by agents in a planning phase - which will provide them with useful context about past decisions and nuanced details about your unique engineering context.
- Code review agents can reference the plan when verifying the implementation.
- What takes time is rarely the code - it's usually the testing and deployment and learnings that come from confronting new code with the harsh reality of users, infra, security, etc. Continuing to iterate on the plan helps future product development by documenting those hard-learned lessons.
- It makes the iterative process much safer, since at any given time you have a materialized version of what the agent is going to build. It forces you to verify each step.
- It will save a lot of context tokens in the future as the agent can ramp up on the topic by just reading the plan, as opposed to reading many code files etc.

## What's in a plan

The outline should look something like this:

````markdown
# TICKET-NUMBER: Feature title

- Status: [approved/implemented/rejected]
- Author:
- Finished on:

Key learnings since this plan was implemented:

- TODO

## Overview

## Requirements

_Short summary of the functional requirements, or link to a PRD (Product Requirements Document). Key edge cases to consider._

## Architecture

(ideally with a schema)

- Current/future state
- Key principles
- Key technical decisions (language, framework, database, libraries) and their tradeoffs
- Dependencies
- Invariants we need to maintain
- Tradeoffs, existing solutions & patterns (with reference to files)
- What's out of scope

## Key implementation details

- Main types and model
- Database structure

## Implementation plan

_Split the implementation in different phases. For each phase:_

- Key tasks
- Success criteria (automated and manual manual tests)

## File structure

```raw
project/
├── src/
│   ├── component1/
│   └── component2/
├── tests/
└── docs/
```

## Rollout

- Rollback plan
- Feature flag usage
- Canary
- Migration (safe roll back)
- Data migration
- Communication plan with users and clients
- Risks (with prioritization and prevention/mitigation strategies)
- Cleanup (e.g., feature flags, docs, performance optimization)
- Future direction

## Other considerations

- Security (e.g., authorization)
- Performance & resource management
- Customer support
- Privacy (e.g., data minimization)
- Testing consideration

## Definition of done checklist

_This section only mentions the aspects relevant for the AI agent_

_P0: must have. P1 can be follow-up items, especially for a prototype_

- [ ] P0 Security: Has appropriate access control
- [ ] P0 Testing: Has appropriate test coverage (incl. unit test)
- [ ] P0 Rollout: Has rollout strategy (e.g., feature flag, canary, communication plan)
- [ ] P0 Observability: Has monitoring
- [ ] P0 Observability: Has alerting (incl. SLO monitoring)
- [ ] P0 Observability: Has user analytics
- [ ] P1 Observability: Has appropriate logging
- [ ] P1 Quality: Has clear and actionable errors
- [ ] P1 Maintainability: Has up-to-date documentation (including runbooks)
- [ ] P1 Quality: Has proper disaster recover (e.g., graceful degradation)
- [ ] P1 Quality: Has internationalization, if applicable (e.g., translations)
- [ ] P1 Rollout: Has risk prevention and mitigation strategies

## References

_List any other relevant plans, RFC, designs, related work, internal docs, packages, tickets._
````

## Plan guidelines

Your agent should reference a document that explain how to use the template above:

```markdown
# Planning a feature

## Plan template

The plan template is at ...

## Checklist before finalizing a plan

Verify:

### Research

- [ ] Looked for similar patterns in the codebase
- [ ] Looked for existing libraries
- [ ] Reviewed relevant plans

### Plan

- [ ] Plan is in the right folder and has the right filename
```

## Tips

- AI-generated text tends to be very repetitive. I summarize and remove repetition from time to time.
- It's totally fine to update the plan as you learn new things during the implementation phase.
- I often ask different models to review the plan and criticize it.
- Have some naming convention for the plan, e.g., "tr-20209-add-toaster-thermostat.md"
- Have rules to define what your plan outline should look like, and other best practices. Or provide a plan template (reference/verify your "definition of done" if you have one)
- Use checklists. Checklist are faster to review, templatize, and maintain.
- Make sure the plan include code examples so that you can get a sense of the final result (especially for introducing new abstractions and patterns).
- Use PR draft mode - review the plan and make sure you agree with everything that's written.
- Use subagents to do extra research on finer points.
- Use schema (with mermaid). Human can focus on those.
- The length of the plan should be commensurate with the complexity of the project.

## Coding agents will increase the momentum towards GitOps

If everything is checked into the repo, coding agents have a much better shot at one-shotting whole projects or features. The more control they have over the end-to-end development workflow, the faster your velocity!

You should consider moving everything into code repositories:

- Infra as code (Terraform/Pulumi)
- Other declarative infra frameworks
- Data frameworks such as dbt or sqlmesh
- Documentation
- Best practices including coding guidelines
- Runbook
- Dashboard and alert configuration
- CMS (content management system)
- Translations

It goes without saying that this doubles as documentation for your team, which will speed up onboarding new engineers.

## Resources

As always, there are more resources on my repo [charlax/professional-programming](https://github.com/charlax/professional-programming)

- [Architectural Decision Records](https://adr.github.io/ad-practices/)
- [Claude Code creator Boris shares his setup with 13 detailed steps](https://www.reddit.com/r/ClaudeAI/comments/1q2c0ne/claude_code_creator_boris_shares_his_setup_with/)
- [Using LLMs at Oxide / RFD / Oxide](https://rfd.shared.oxide.computer/rfd/0576)
- [Getting AI to Work in Complex Codebases](https://github.com/humanlayer/advanced-context-engineering-for-coding-agents/blob/main/ace-fca.md)
- [Writing Helpful Error Messages](https://developers.google.com/tech-writing/error-messages), Google for Developers
- [Specifications Are Becoming the Real Source Code](https://7ynk3r.github.io/2026/01/19/specifications-are-becoming-the-real-source-code/)
- [How I Use Claude Code](https://boristane.com/blog/how-i-use-claude-code/)
- [A sufficiently detailed spec is code](https://haskellforall.com/2026/03/a-sufficiently-detailed-spec-is-code)

## Thanks

Thanks to my colleague Sean Koop for providing the inspiration for the practices described in this article.


Why you should start your AI coding session with planning mode, and check out the resulting plan

Write and checkout the plan


When building a new project, it's a smart move to be very strict right from the start. It is much harder to add more linting/typing checks once you have 1000+ lines of code.

That's why I'm providing an opinionated list of libraries for your new Python project. I might write a more in-depth article on the best practices when building a web app with Python. For now, this is mostly a checklist with some obvious recommendations.

**Why not a template repo instead of a checklist?** Template repositories (e.g. built with [cookiecutter](https://github.com/cookiecutter/cookiecutter)) go quickly out of date and discourage learning about the ins and outs of all those best practices. They might make sense for your organization, but they're not the point of this article.

## Summary

- Development tasks: `Makefile`
- Typechecking: [mypy](http://mypy-lang.org/)
- Dependency and virtualenv management: [poetry](https://python-poetry.org/)
- Linting: [flake8](https://flake8.pycqa.org/en/latest/)
- Code autoformatting: [black](https://github.com/psf/black) and [isort](https://pycqa.github.io/isort/)
- Tests: [pytest](https://docs.pytest.org/en/stable/) with plugins such as
  [pytest-cov](https://github.com/pytest-dev/pytest-cov), [pytest-factoryboy](https://pytest-factoryboy.readthedocs.io/en/latest/)
- Docstring: [pydocstyle](http://www.pydocstyle.org/en/stable/)
- Logging: [structlog](https://www.structlog.org/en/stable/)
- Configuration: [Pydantic BaseSettings](https://pydantic-docs.helpmanual.io/usage/settings/) with [dotenv support](https://pypi.org/project/python-dotenv/)
- Error reporting: [Sentry](https://sentry.io/welcome/)
- Documentation: [Docusaurus](https://docusaurus.io/) or [Sphinx](https://www.sphinx-doc.org/en/master/)
- Profiling and performance: [profile](https://docs.python.org/3/library/profile.html), [pyinstrument](https://github.com/joerick/pyinstrument), [sqltap](https://github.com/inconshreveable/sqltap)

Other topics:

- Data validation: [Pydantic Models](https://pydantic-docs.helpmanual.io/usage/models/)
- DB ORM: [sqlalchemy](https://www.sqlalchemy.org/docs/latest/) and [alembic](https://alembic.sqlalchemy.org/) for migrations
- Web framework: [fastapi](https://fastapi.tiangolo.com/)
- FP utilities: [toolz](https://toolz.readthedocs.io/)
- CLI framework: [typer](https://typer.tiangolo.com/)

Not detailed below:

- [pre-commit](https://pre-commit.com/)

## Running development tasks: Makefile

Makefiles are well understood, work almost everywhere, and shorten the ramp-up time for fellow developers who might not have much experience with Python.

Here's an example:

```makefile
SHELL := bash
.ONESHELL:
.SHELLFLAGS := -eu -o pipefail -c
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules

install:  # Install the app locally
	poetry install
.PHONY: install

ci: typecheck lint test ## Run all checks (test, lint, typecheck)
.PHONY: ci

test:  ## Run tests
	poetry run pytest .
.PHONY: test

lint:  ## Run linting
	poetry run black --check .
	poetry run isort -c .
	poetry run flake8 .
	poetry run pydocstyle .
.PHONY: lint

lint-fix:  ## Run autoformatters
	poetry run black .
	poetry run isort .
.PHONY: lint-fix

typecheck:  ## Run typechecking
	poetry run mypy --show-error-codes --pretty .
.PHONY: typecheck

.DEFAULT_GOAL := help
help: Makefile
	@grep -E '(^[a-zA-Z_-]+:.*?##.*$$)|(^##)' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[32m%-30s\033[0m %s\n", $$1, $$2}' | sed -e 's/\[32m##/[33m/'
```

This is a great article about Makefile: [Your Makefiles are wrong](https://tech.davis-hansson.com/p/make/)

## Typechecking: mypy

Type annotations in Python libraries are not yet pervasive, but getting better
every day.

Supported by Python's founder, Guido van Rossum, [mypy](http://mypy-lang.org/)
is the de facto standard. The [cheat sheet](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html)
is a very helpful resource.

You can configure mypy inside `setup.cfg`:

```toml
[mypy]
strict = true

# I prefer to be explicit about ignoring packages which do not yet have types:
[mypy-psycopg2.*]
ignore_missing_imports = True
```

## Dependency and virtualenv management: poetry

Unfortunately because of the way `pip` installs dependencies, you have to deal
with virtualenv in most cases (although [things might change rapidly with pdm](https://pdm.fming.dev/)).

Nowadays I'd recommend using [poetry](https://python-poetry.org/). It is not
yet absolutely perfect, but it provides a very elegant CLI API.

It's super easy to start a project:

```bash
poetry new service-name
cd service-name
$EDITOR pyproject.toml
rm -Rf tests
mv README.rst README.md

poetry add sqlalchemy  # for instance
```

Then either go into a virtualenv-enabled shell with `poetry shell` or prefix
your commands with `poetry run ...`.

## Linting: flake8

There are two main linters:

- [flake8](https://flake8.pycqa.org/en/latest/)
- [pylint](https://www.pylint.org/)

I usually rely mostly on flake8 because it has fewer false positives. While
pylint is super configurable, it includes too many checks to my taste.

I use the following configuration (in `setup.cfg`):

```toml
[flake8]
max-line-length = 99
extend-ignore =
    # See https://github.com/PyCQA/pycodestyle/issues/373
    E203,
```

## Code autoformatting: black and isort

[black](https://github.com/psf/black) autoformats your code so that you don't
have to think about it.

[isort](https://pycqa.github.io/isort/) is a nice addition to black, it will
sort your imports to comply with [PEP 8](https://www.python.org/dev/peps/pep-0008/#imports):

- Sort alphabetically
- Group into standard imports, third-party imports, app imports

Here's my isort config to comply with flake8 and black (in
`pyproject.toml`):

```toml
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
line_length = 88
```

## Test: pytest with coverage

[pytest](https://docs.pytest.org/en/stable/) has a lot of magical features but
it makes writing tests so efficient. The fixture system is brilliant and
super powerful. Using plain `assert` instead of having to learn an `assertEqual`
metalanguage makes your life more meaningful.

Here's the config I use (in `pyproject.toml`):

```toml
[tool.pytest.ini_options]
# Personal preference: I am too used to native traceback
addopts = "--tb=short"

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "def __repr__",
    "if __name__ == .__main__.:",
    "nocov",
    "if TYPE_CHECKING:",
]

[tool.coverage.run]
# Activating branch coverage is super important
branch = true
omit = [
  # add your files to omit here
	]
```

I usually use the following plugins and lib:

- [pytest-factoryboy](https://pytest-factoryboy.readthedocs.io/en/latest/) use
  factories to create your fixture. Super powerful, and avoids having a single
  file where all your reusable fixtures are defined a thousand times with
  different variations.
- [pytest-mock](https://github.com/pytest-dev/pytest-mock/) makes it easier to
  work with `unittest.mock`.
- [pytest-cov](https://github.com/pytest-dev/pytest-cov) provides coverage
  reports for your tests.
- [doubles](https://github.com/uber/doubles): sadly not maintained anymore (but
  it still works!), it provides a much simpler and stricter mocking experience
  than `unittest.mock`.
- [requests-mock](https://requests-mock.readthedocs.io/en/latest/) to check
  integration with HTTP services called with [requests](https://requests.readthedocs.io/en/master/). It automatically integrates with pytest and provides a fixture named `requests_mock`.

## Checking docstring: pydocstyle

[pydocstyle](http://www.pydocstyle.org/en/stable/) enforces [PEP
257](https://www.python.org/dev/peps/pep-0257/) for docstring styling.

Here's my config (in `setup.cfg`):

```toml
[pydocstyle]
# Do not require any docstring
ignore = D100,D101,D102,D103,D104,D105,D106,D107,D213,D203
```

## Logging: structlog

[structlog](https://www.structlog.org/en/stable/) is a must-have for all your
logging needs.

```python
from structlog import get_logger


logger = get_logger(__name__)

def hello(name: str):
		logger.info("saying hello", name=name)
		# instead of :
		# logger.info("saying hello to %s", name)
```

Structuring your logs has numerous advantages:

- Immediately parsable by automated tools (kibana, m/r jobs, etc.)
- Easier to write: you don't have to think about the order of your logging
  message
- Flexible: can be further manipulated since all log messages are dicts
  until they're displayed

You can create `yourapp.lib.log`:

```python
import json
import logging
from typing import Any, Dict
from uuid import UUID

import structlog

from yourapp.config import config


def default(obj: Any) -> Any:
    if isinstance(obj, UUID):
        return str(obj)

    raise TypeError(f"Can't serialize {type(obj)}")


def dumps(*args: Any, **kwargs: Any) -> str:
    kwargs.pop("default", None)
    return json.dumps(*args, **kwargs, default=default)


def add_version(
    logger: logging.Logger, method_name: str, event_dict: Dict[str, Any]
) -> Dict[str, Any]:
    """Add version to log message."""
    event_dict["version"] = config.git_commit_short
    return event_dict


class ConsoleRenderer(structlog.dev.ConsoleRenderer):
    def _repr(self, val: Any) -> str:
        # Display shorter uuid
        # https://www.structlog.org/en/stable/_modules/structlog/dev.html#ConsoleRenderer
        if isinstance(val, UUID):
            return str(val)
        return super()._repr(val)


def configure_logger(level: str = "INFO", *, console: bool = False) -> None:
    """Configure logging.

    console should be True for console (dev) environment.
    """
    # see https://stackoverflow.com/questions/37703609/using-python-logging-with-aws-lambda
    root = logging.getLogger()
    if root.handlers:
        for handler in root.handlers:
            root.removeHandler(handler)
    logging.basicConfig(format="%(message)s", level=level)

    if not console:
        processors = [
            add_version,
            structlog.stdlib.filter_by_level,
            structlog.stdlib.add_logger_name,
            structlog.stdlib.add_log_level,
            structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M.%S"),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            structlog.processors.JSONRenderer(serializer=dumps),
        ]
    else:  # nocov
        processors = [
            structlog.stdlib.add_logger_name,
            structlog.stdlib.add_log_level,
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.TimeStamper(fmt="%Y-%m-%d %H:%M.%S"),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            ConsoleRenderer(),
        ]

    structlog.configure(
        processors=processors,  # type: ignore
        wrapper_class=structlog.stdlib.BoundLogger,
        logger_factory=structlog.stdlib.LoggerFactory(),
        cache_logger_on_first_use=True,
    )
```

## Configuration: Pydantic's BaseSettings

Like most people, I usually ended up having my own mechanism for handling
configuration. Thanks to the web framework fastapi, I've discovered that
[pydantic](https://pydantic-docs.helpmanual.io/) provides a very handy
[BaseSettings](https://pydantic-docs.helpmanual.io/usage/settings/) class that
relies on environment variable for its configuration. `BaseSettings` provide
many things that would be annoying to implement from scratch:

- Type hints
- Read from environment variables
- Validate configuration values
- `.env` support with [python-dotenv](https://pypi.org/project/python-dotenv/)
- Secrets support

```python
import os
from pathlib import Path
from typing import List, Optional

from dotenv import load_dotenv
from pydantic import BaseSettings

ENV_FILENAME = os.environ.get("DOTENV", ".env")


class MisconfiguredException(Exception):
    pass


class Config(BaseSettings):
    # Please use env_name ONLY for informational purpose (see docs)
    env_name: str
    git_commit_short: str = "unknown"

    # Activate this to get profiling - see documentation.
    is_db_enabled: bool = False

    db_user: str = "unconfigured"
    db_password: str = "unconfigured"
    db_name: str = "unconfigured"
    db_port: str = "5432"
    db_host: str = "localhost"

    sentry_dsn: Optional[str]


def get_config() -> Config:
    """Get the config."""
    # We follow serverless's dotenv plugin's behavior here:
    # https://www.npmjs.com/package/serverless-dotenv-plugin

    # First load .env
    load_dotenv(dotenv_path=".env")

    if not Path(ENV_FILENAME).exists():
        raise ValueError(f"Config file {ENV_FILENAME} does not exist.")

    if ENV_FILENAME.endswith(".local"):
        raise ValueError(
            "Expected env filename like '.env.dev', "
            f"got override ending with .local instead: {ENV_FILENAME!r}. "
            f" Try with {ENV_FILENAME.replace('.local', '')!r}"
        )

    # Then load .env.{env}
    load_dotenv(dotenv_path=ENV_FILENAME)

    # Then load .env.{env}.local if it exists
    override = ENV_FILENAME + ".local"
    if Path(override).exists():
        load_dotenv(dotenv_path=override)

    return Config()


config = get_config()
```

Now you just have to run your commands like this:

```bash
DOTENV=.env.test poetry run pytest .
```

## Error reporting: Sentry

[Sentry](https://sentry.io/welcome/) is a service that provides exception
monitoring. Its SDK is very simple to integrate.

Usually, I use the following pattern:

```python
# app.py
from app.lib.log import configure_logger
from app.lib.sentry import configure_sentry

configure_logger(config.log_level)
configure_sentry()
```

You need a route to test the integration (example below is for Flask):

```python
@app.server.route("/health/sentry", methods=["GET"])
def get_sentry_error() -> str:
    """Raise and report an exception to test Sentry integration."""

    class TestSentryException(Exception):
        pass

    try:
        raise TestSentryException("Testing Sentry integration")
    except TestSentryException as exc:
        sentry_sdk.capture_exception(exc)

    return "Reported Sentry errror"
```

```python
# app.lib.sentry
from typing import Any

import sentry_sdk
from structlog import get_logger

from app.config import config

logger = get_logger(__name__)


# This example is for Flask, adapt to your HTTP framework
@app.route("/health/sentry", methods=["GET"])
def get_sentry_error() -> str:
    """Raise and report an exception to test Sentry integration."""
    logger.info("info: testing Sentry integration")


    class TestSentryException(Exception):
        pass

    try:
        raise TestSentryException("Testing Sentry integration")
    except TestSentryException as exc:
        sentry_sdk.capture_exception(exc)

    return "Reported Sentry error"
```

## Documentation: Docusaurus

[Docusaurus](https://docusaurus.io/) is a powerful yet simple documentation management tool.

[Sphinx](https://www.sphinx-doc.org/en/master/) is another great choice (especially if you want to get Python code auto-documentation).

## Domain models, data validation: Pydantic

[Pydantic Models](https://pydantic-docs.helpmanual.io/usage/models/) provides a flexible way to define your domain model objects:

- Type hinting
- Validators
- Export to json-schema

You can also use Python's standard lib [dataclass](https://docs.python.org/3/library/dataclasses.html) together with something like [marshmallow](https://marshmallow.readthedocs.io/en/stable/).

## ORM: sqlalchemy

If you need to interact with the DB, [sqlalchemy](https://www.sqlalchemy.org/docs/latest/) is a very safe choice. It comes with loads of features and is the most used non-Django Python ORM, which means that you'll find Stack Overflow solution for all your problems. Using [alembic](https://alembic.sqlalchemy.org/) for DB migrations is the next logical move.

Both libraries are written by the insanely productive [Mike Bayer](https://github.com/zzzeek).

## Web framework: fastapi

You have a lot of excellent choices when it comes to Python web framework.
I usually prefer microframework and am currently developing with
[fastapi](https://fastapi.tiangolo.com/), which is a lot of fun to work with.

I usually refrain from using any plugins that come with the framework, because
they are usually too coupled to the framework and the context of an HTTP
request. I might write an article about my preferred setup.

## Utility functions for functional programming: toolz

While Python is not a strict functional programming language, it is possible to
write FP-styled code with it.

[toolz](https://toolz.readthedocs.io/) is a great companion and provides many utility functions that make writing code easier. It has a curried-by-default namespace (`from toolz.curried import take`). Checkout its [cheatsheet](https://toolz.readthedocs.io/en/latest/api.html).

## CLI framework: typer

[typer](https://typer.tiangolo.com/) (same author as fastapi and pydantic) leverages type annotations to make it super easy to write powerful scripts with a command line interface:

```python
#!/usr/bin/env python3

"""Say hello.
"""

import typer


def main(name: str) -> int:
    typer.echo(f"Hello {name}")

    password = typer.prompt("password")
    assert len(password) > 8

    return 0


if __name__ == "__main__":
    typer.run(main)
```

## Performance profiling: pyinstrument and sqltap

- [pyinstrument](https://github.com/joerick/pyinstrument) is a recent Python profiler which can export to HTML.
- [sqltap](https://github.com/inconshreveable/sqltap) integrates with sqlalchemy to allow you to introspect SQL queries and also exports to HTML.

Some services (like Sentry) can also profile code thanks to their SDK.
Otherwise, you can also rely on the standard library module [profile](https://docs.python.org/3/library/profile.html).

## Wishlist

- Lots of inconsistencies for where to put configuration: `setup.cfg`,
  `pyproject.toml`, specific files, etc.
- The more complex the library, the more useful it would be to have type
  annotations and... the less probable it is to have those annotations.
  Libraries such as sqlalchemy (coming in 2.0) and toolz don't have official types for now.
- mypy encourages nominal subtyping (see [this FAQ](https://mypy.readthedocs.io/en/stable/faq.html#can-i-use-duck-typing-with-mypy)) which is a bit sad because it discourages using simple dicts. Fortunately something like [PEP 589 TypeDict](https://www.python.org/dev/peps/pep-0589/) will improve things.
- It would be so nice to avoid having to use virtualenv (even through
  something like poetry or pipenv). First-time Python developers get so
  confused about them (compared to Node's simpler `node_modules` setup). I'm really looking forward to seeing [PEP
  582](https://www.python.org/dev/peps/pep-0582/) deployed.

## Missing something?

Drop me an email at `charles at dein.fr` if you think I'm missing something!

For more resources related to Python, check out my repo
[charlax/python-education](https://github.com/charlax/python-education).


All the best practices for setting up a new app in Python

A Python project checklist


Functional programming (FP) has been picking up steam those last few years. It
is now the de facto standard for front-end programming with frameworks such as
React, which leverages a lot of its concepts (higher-order components,
immutability, reducers, avoiding object-oriented programming constructs
such inheritance, etc.).

As a pragmatic programmer who's not an expert in FP, I'm taking more
and more inspiration from FP to write code in my day-to-day languages (Python
and JavaScript/TypeScript, mostly). While those languages are not pure FP
languages (e.g., they don't have strict support for immutability,
currying by default), they are sufficiently versatile and pragmatic that you
can absolutely write using a functional _style_.

## Object-oriented patterns to avoid

OOP (object-oriented programming) has its use cases. If you find yourself having to reinvent concepts from OOP with functions, you're probably doing it wrong and you should probably reuse battle-tested OOP patterns instead.

That being said, there are a number of OOP patterns and style to avoid. Here are two of them:

- Avoid inheritance
- Avoid modifying instance state

### Avoid inheritance

Avoiding inheritance is a pretty old idea, notably popularized by the Gang of
Four book ([Design Patterns: Elements of Reusable Object-Oriented
Software](https://en.wikipedia.org/wiki/Design_Patterns) published in 1994),
but for some reason it still very present in lots of frameworks.

The React documentation explains it pretty well in [why composition makes more sense than inheritance for UI](https://reactjs.org/docs/composition-vs-inheritance.html).

Multiple layers of inheritance leads to code that is very complicated to follow. This is even more the case when you use abstract classes and have some method to override and some with default implementation. Over more than one layer of inheritance, understanding the flow is a high-cognitive load effort.

Let's say you have this "abstract" class:

```python title="abstract_toaster.py"
class AbstractToaster:
    def set_heat(self):
        raise NotImplementedError()

    def heat(self):
        self.set_heat(4)

    def toast(self):
        self.heat()
        self.wait(4)
        self.set_heat(0)
```

And this concrete implementation:

```python title="toaster.py"
class Toaster(AbstractToaster):
    def set_heat(self):
        ...

toaster = Toaster()
toaster.toast()
```

In `toaster.py` it is quite impossible to understand what's going on, and you'll have to open `abstract_toaster.py` to understand the flow. This is fine with one layer and two files. It becomes a nightmare when there are multiple levels, a larger number of more complicated methods, and multiple superclasses.

### Avoid modifying instance state

Let's take this extreme example which creates a class to sum up list:

```python
class Adder:
    def __init__(self, values: list[int]):
        self.values = values
        self._current_value = None

    def get_sum(self) -> int:
        """Get the sum of a list."""
        self._sum = 0
        for value in self.values:
            self._current_value = value
            self._add()

        return self._sum

    def _add(self) -> None:
        """Add the current value to the running sum."""
        self._sum += self._current_value


result = Adder([1, 1]).get_sum()
assert result == 2
```

Granted - this is an extreme example. But you might see a similar structure with state stored on the instance, modified and queried in different places. This code will prove very complicated to test, because you need to take into account the order in which things are supposed to be called.

Avoiding global variables is well accepted. They make code difficult to reason about and to test. Note that the previous example is actually very similar to global state:

```python
_CURRENT_SUM = None

def get_sum(values: list[int]) -> int:
    """Get the sum of a list."""
    global _CURRENT_SUM
    _CURRENT_SUM = 0

    for value in values:
        _CURRENT_SUM += value

    return _CURRENT_SUM


result = get_sum([1, 1])
assert result == 2
```

What are the issues with modifying instance state?

- It's more difficult to read: because the state `self._sum` comes from outside the function, so that's one more thing your brain has to keep track of.
- It's more difficult to follow: there are no guarantee that `self._sum` is not modified somewhere else. In a real world example with subclasses, this becomes a nightmare.
- It's more difficult to test: you have to setup the `Adder` instance before testing. In a real world example, this might take a few lines.

## Take inspiration from FP constructs

You can take advantages of FP even in languages that are not considered pure FP languages. Even though you won't get strict guarantees, it will help you write more readable & testable code.

### Limiting and isolating side effects

A function is said to have side effects if it causes observable changes in a system's state in addition to (or instead of) its return value.

```python
def no_side_effect(name: str) -> str:
    return f"hello, {name}"


# This function prints to screen, which is a side effect
def with_side_effect(name: str) -> None:
    print(f"hello, {name}")
```

Another way to think about side effects is testing. If asserting the return value of the function is sufficient to validate its core behavior, then it does not have any side effect:

```python
# To check the function's behavior, we only need to check its return value
def test_no_side_effet():
    assert no_side_effect("Louis") == "hello, Louis"


# In this test we have just checked the return value, but we still need to
# check that it has printed something on the screen. This test is incomplete.
def test_with_side_effet():
    assert with_side_effect("Louis") is None
```

Here are some examples of side effects:

- Reading/writing the database
- Reading/writing a file
- Printing something on the screen
- Requesting an HTTP service
- Getting feedback from the user

Evidently, you must have side effects at some point. So the key idea here is to cleanly set aside code that has side effect, from pure functional code.

If you follow the Domain Driven Design mindset, this can be done by having all side effects in specialized layers, such as Repository, Gateway, etc. Then you can strive for side effect-free domain services.

Side-effect free functions are easier to test, easier to read and easier to debug, so they should be preferred whenever you can. Try to isolate the side effects as much as possible:

Let's say we have a list of image urls and we want to filter out the images that are too small. Here's a version that does not cleanly isolate side effects:

```python
import requests

MIN_SIZE = 1024 * 1024


def filter_out_small_images(image_urls: list[str]) -> list[str]:
    returned = []
    for url in image_urls:
        # this makes an HTTP request
        res = requests.get(url)
        assert res.status_code == 200
        if len(res.content) > MIN_SIZE:
            returned.append(url)

    return returned
```

Testing this function is going to be annoying, because we are making HTTP
requests in the middle of it. Since we should avoid external dependencies in
unit tests, we'll have to mock out the HTTP request, which will involve a bit
of code. This function accepts a list, so it might make multiple HTTP calls
that we'll have to mock - which is even more complicated.

Another thing we can observe is that this code is not as readable as it could
be. The true business logic (filtering out small images) is hidden in the
middle of HTTP request logic.

So the first step is to create a `get_size` function that has the side effect.
We still need to wire this with the rest of the logic. Note that as an added
benefit, `get_size` is more generic and can be reused for other URLs.

```python
import requests


def get_size(url: str) -> int:
    """Get an URL's size."""
    res = requests.get(url)
    assert res.status_code == 200
    return len(res.content)
```

### Embrace a declarative style

While imperative code tells the computer what to do, declarative code tells it
what we want. An imperative style usually mixes implementation details with
core business logic, while declarative usually expresses pure intent.

The best example of declarative code is SQL. When we write SQL, we don't tell
it how to return results, we just tells it what we want:

```sql
select brand, count(id) from products where price > 100 group by brand order by brand
```

Unfortunately we can't (yet) write code like this - but we can use a similar
style, where we isolate the imperative code and write business logic using
a higher level language.

If you look at `filter_out_small_images`, we use an imperative style: we create
an empty list, we iterate over image URLs and we append to the list when the
criteria is met. Luckily there are two FP constructs we can use to make this
more declarative:

- `map(func, iterable)` returns an iterable `func(iterable[0]), func(iterable[1]), ...`
- `filter(func, iterable)` returns an iterable for which `func(item)` returns true.

```python
import requests

MIN_SIZE = 1024 * 1024


def get_size(url: str) -> int:
    """Get an URL's size."""
    res = requests.get(url)
    assert res.status_code == 200
    return len(res.content)


def is_large(url: str) -> bool:
    return get_size(url) > MIN_SIZE


def filter_out_small_images(image_urls: list[str]) -> list[str]:
    # We use list() because filter() returns an iterable, not a list
    return list(filter(is_large, image_urls))
```

If your programming language supports list comprehension, you can use them for
such a simple example (personal taste!):

```python
def filter_out_small_images(image_urls: list[str]) -> list[str]:
    return [url for url in image_urls if is_large(url)]
```

As you can see our code is getting even more cleanly separated, with small
functions doing simple things that are easy to test. We still have to deal with
the side effect though.

This code is also less imperative since we use `filter`, which
guarantees that we are manipulating iterables and returning iterables. Once you
are used to `map`, `filter`, `reduce`, etc. you immediately know what kind of
guarantees you get. For instance with `filter`:

- The input must be an iterable.
- The output is an iterable.
- The output is at most the same length as the input iterable.
- The items in the output were all in the input.

All those guarantees conspire to make your code easier to reason about, just by
using `filter`. Compare to this simplistic example:

```python
def filter_out_small_images(image_urls: list[str]) -> list[str]:
    # It seems we're going to return a list
    returned = []
    for url in image_urls:
        if get_size(res.content) > MIN_SIZE:
            # We are appending the URL if it is large enough
            returned.append(url)

    return returned
```

Using the imperative style, you have to actually read the code to understand
what's going on.

### Use bags of data

> It's better to have 100 functions operate on one data structure than 10
> functions on 10 datastructures
> -- Alan Perlis (quoted in Clojure for the brave and true by Daniel Higginbotham)

As we have stated, we are still doing side effects in the
middle of the business logic. It would be nice if we could separate them more
cleanly, and have filter manipulate simple bags of data. For the sake of the
example, if you are used to OOP, you might want to write things like this:

```python
import requests


class ImageUrl:
    def __init__(self, url: str):
        self.url = url

    def get_size(self)
        res = requests.get(self.url)
        assert res.status_code == 200
        return len(res.content)


def is_large(url: ImageUrl) -> bool:
    return url.get_size() > MIN_SIZE


def filter_out_small_images(image_urls: list[ImageUrl]) -> list[ImageUrl]:
    return list(filter(is_large, image_urls))
```

This is even worse than what we had in the beginning:

- Our `get_size` logic is tied to the `ImageUrl` object and can't be reused.
- We are making an HTTP request inside an object (side effect).

For such a simple example, it might look fine, but we can do better. Look at
the following example:

```python
from dataclasses import dataclass


@dataclass
class UrlWithSize:
    url: str
    size: int


def get_url_with_size(url: str) -> UrlWithSize:
    size = get_size(url)
    return UrlWithSize(url, size)
```

## To be continued...

Consider this a short introduction to FP! This shows that FP leads to simpler code that is easier to test. Happy coding!

- Simple functions that can be composed. Each function can be tested in isolation.
- Classes do have their use cases in certain languages. It's usually a good idea to keep inheritance to one level only.
- FP is stricter, e.g. `map` guarantees same number of values
- Functions require less ceremony than OOP

## References

- Charles Scalfani, [Goodbye, Object Oriented Programming](https://medium.com/@cscalfani/goodbye-object-oriented-programming-a59cda4c0e53#.39ax09e4k)
- Matthew Gerstman, [Functional Programming Fundamentals](https://www.matthewgerstman.com/tech/functional-programming-fundamentals/)
- Robert C. Martin, [OO vs FP](https://blog.cleancoder.com/uncle-bob/2014/11/24/FPvsOO.html)
- Section "Functional Programming" on [charlax/professional-programming](https://github.com/charlax/professional-programming#functional-programming-fp)


Taking inspiration from FP patterns leads to simpler code, even in languages that don't fully support them.

A functional style leads to simpler code


**Summary**: all lines of code should be covered by a test or _explicitly_
excluded from test coverage.

What's your test coverage? What do you consider a good test coverage
percentage?

Those questions often get asked during engineering interviews.
It gives the interviewee a sense of how much the company cares about quality.
It's also a good question for the candidate because you can't have a cargo
cult answer. You need to argue. It gives useful insights into one's quality
mindset.

I believe the only right value to track for a test coverage metric is 100% of
the _lines that should be covered_ (_not_ the lines that are written), which
ensures that the lines that you don't want to cover are explicit, and not
implicit.

Please don't skim the articles as you'll find that the point I'm making is pretty common sense. If you still disagree have a look at the response
to common objections at the end.

_Note: I'm using Python for all the examples below, but the principles are
applicable regardless of the programming language._

## Why you should measure test coverage

If you're not already monitoring test coverage, you should probably start now
(if you don't have tests, that's for another article). Some programming
languages have better support than others. Python has excellent support for it,
through `pytest` and `pytest-cov` (using `coverage` under the hood).

### Test all functionality

Test coverage allows you to check which lines of code are run while the tests
are executed. If some lines are missing coverage, it means you're not testing
("covering") the underlying behavior. You could introduce a bug, and no test
would fail, which is not desirable. Introducing a bug should ideally trigger a
test to fail.

Let's see a contrived example:

```python
def toast(bread: str) -> str:
    if bread == "brioche":
        ...
        1 / 0  # let's say a bug is hidden here
        return "ok"

    return bread

def test_toast():
    assert toast("baguette") == "ok"
```

The test will pass, even though `toast("brioche")` should fail. Adding
a test for this case will let you find the `ZeroDivisionError`.

### Ensure all tests are run

Another case for checking is that you want to make sure that your test actually
run. In this contrived example, the tests will pass, even though one of
those test does not do anything:

```python
def identity(bread: str) -> str:
    return bread

def test_identity():
    assert identity("brioche") == "brioche"

def test_identity_noop():
    if 1 == 1:  # this is a contrived example, there shouldn't be any 'if' in tests
        return
    assert identity("baguette") == "baguette"
```

Once you've agreed that you need test coverage, the question remains: what is the
right level of coverage? It is very costly to ensure that absolutely every line
of code is tested. Should it be 80%? 90%?

## Why you should check for 100% coverage

```text
TOTAL                                      18435     16   1742      0    99%

FAIL Required test coverage of 100% not reached. Total coverage: 99.86%
```

100% is clear cut.

Enforcing 100% coverage requires you to be explicit about the lines that don't
have to be covered ("explicit is better than implicit"). By including the `no cover` marker (or whatever your language/coverage lib supports) next to the
code that is excluded from coverage, you allow code reviewers to see your
coverage decisions, and challenge them.

The problem with having a goal that is not 100% is that it leaves room for
interpretation and negotiation. If we're currently at 80% and I bring coverage
at 79.9% because I'm too lazy to test my code (or exclude it from coverage),
then can't we say that it's ok? Explicit markers in the code make the
discussion more factual and objective.

The other problem with something different from 100% is that you might be
adding coverage in one area, and reducing it in another, without knowing it. It
might still lead to 80% coverage - but the reduction in coverage might have
appeared in a crucial area of your code.

## Will 100% coverage prevent any bug from being introduced?

Of course not. Just because a line is run in a test does not mean it is 100%
correct. You might not be testing all variations. The test might assert the
wrong thing.

Some lines of code should be tested twice to ensure they're correct. See this
contrived example:

```python
def can_toast(bread: str) -> bool:
    return bread in ["croissant"]  # this line is covered

def test_can_toast():
    assert can_toast("baguette") is True
```

All the lines are covered, but I'm not testing that you can't toast `"brioche"`
and `"baguette"`. Let's say this behavior is critical (toasting a croissant is
a crime!). Without another test, somebody could change the code like this, and
nothing would break.

```python
def can_toast(bread: str) -> bool:
    return True  # this line is still covered

def test_can_toast():
    assert can_toast("baguette") is True
```

Therefore sometime you definitely should test the same line more than once,
testing the failure case and the success case. I'd strongly recommend this for
permission checking for instance.

```python
def test_can_toast():
    assert can_toast("baguette") is True

def test_can_toast_cannot():
    assert can_toast("croissant") is False
```

### Is this realistic?

You don't need to cover 100% of the lines you write. Some are not worth
covering as you'll see in the examples below. However, you need to cover 100% of the
lines **you want to be covered**. You need to be explicit about excluding lines
from coverage. Tools such as Python's `coverage` allow you to specify which
line should be excluded from the coverage.

Using `pytest-cov` you can achieve this pretty simply:

```python
PLUGGED = True

def toast(bread: str) -> str:
    if not PLUGGED:  # pragma: no cover
        return "ok"

    ...
    return "ok"

def test_toast():
    assert toast("bread") == "ok"
```

With this setup, test coverage will be at 100% even though we did not test the
case where the toaster is not plugged because it is too simple. Writing a test
for this "functionality" would not be worth it. Therefore we explicitly exclude
it from coverage. You should revisit this decision if it becomes more complex
and needs to be tested.

## What can/should be excluded from coverage?

It depends on your codebase and your quality expectations. Also, as
for any rules, there are exceptions. Examples are writing using Python, but this
is not language-dependent.

### Simple error handling

```python
def toast(bread: str) -> str:
    if not bread:  # pragma: no cover - the handling is too obvious
        raise ValueError("no bread provided")
    ...
```

Anything more complicated than that should probably be covered.

### Proxy functions

```python
from external_lib import Toaster

def toast(bread: str) -> str:  # pragma: no cover - we're just proxying
    return Toaster.toast(bread)
```

It also applies to thin wrappers you might have in front of your
dependencies.

### Code as configuration

```python
def get_temperature(bread: str) -> int:  # pragma: no cover - no complexity there
    if bread == "brioche":
        return 1

    elif bread == "baguette":
        return 3

    ...
    return 2
```

### Simple early return/continue

```python
def toast(breads: List[str]) -> str:
    for bread in breads:
        if not bread:  # pragma: no cover - simple early continue
            continue

        ...

    return "ok"
```

If the early return/continue is part of the logic, or leads to complicated side
effects, it should be tested.

### Code that is too difficult to test (rare)

In rare cases, it is a perfectly fine decision to exclude some code from
coverage if the tests are too expensive to write (because of the required
setup, or the use of an external library, for instance). Again - this should
only be in rare cases, for instance when you're using an external library that
is not easy to mock. In most cases, if your code is complicated to test that
probably means it needs to be refactored - easy-to-test code is usually better
designed.

```python
from packagea import A
from packageb import B

def toast(bread: str) -> str:  # pragma: no cover - too complicated to test
    a = A.expensive_function_that_needs_to_be_mocked()
    b = B.expensive_function_that_needs_to_be_mocked()
    ...  # lots of code
    return "ok"
```

Make sure that this code is easy to test manually, and that the code calling it
is well tested.

### Debugging/manual test code

```python
import os

DEBUG = os.environ.get("DEBUGGING")
p
def toast(bread: str) -> str:
    if DEBUG:  # pragma: no cover - debugging code
        print(f"toasting {bread}")
        ...
    return "ok"
```

## What should be covered

Here's a list of things that should be covered, ideally more than
once.

### Branches

If your language supports it, make sure you test all branches:

```python
def toast(bread: str) -> str:        # 1
    if bread == "baguette":          # 2
        bread = "toasted_baguette"   # 3
    return bread                     # 4


def test_toast_baguette():
    assert toast("baguette") == "toasted_baguette"
```

You're technically at 100% code coverage, but the case where `bread` is not
"baguette" and the `if` is never evaluated to False: line 2 never jumps to line 4. If you activate branch coverage, it will be flagged as a partial branch, and
coverage won't be at 100%. To fully test it, you'll need:

```python
def toast(bread: str) -> str:        # 1
    if bread == "baguette":          # 2
        bread = "toasted_baguette"   # 3
    return bread                     # 4


def test_toast_baguette():
    assert toast("baguette") == "toasted_baguette"  # 2 -> 3 -> 4

def test_toast_other():
    assert toast("brioche") == "brioche"  # 2 -> 4
```

As shown in this example, this lets you test more functionality, thus reducing
the probability of bugs.

### Security: permission check, input sanitization...

Anything that relates to security should be heavily tested, with all sorts of
inputs.

```python
import re

import pytest

REGEX = re.compile("[a-z]+")

def toast(bread: str) -> str:
    match = REGEX.match(bread)
    if not match:
        return ValueError("Invalid bread variant")
    ...


def test_toast_fail():  # note: you can use pytest's parametrize feature here.
    with pytest.raises(ValueError):
        toast("12")
    with pytest.raises(ValueError):
        toast("@#*$")
    with pytest.raises(ValueError):
        toast("<script>window.alert('hello world!')</script>")
    with pytest.raises(ValueError):
        toast("'; DROP TABLE users;")

```

### Complex error handling

If error handling is sophisticated enough and can be considered a feature (e.g.
recovering gracefully from some error) - it needs to be covered.

```python
import pytest
from doubles import expect

def toast(bread: str) -> None:
    if bread == "brioche":
        toaster.unplug()
        toaster.cleanup()
        raise ValueError()
    ...

def test_toast_fail():
    expect(toaster).unplug()
    expect(toaster).cleanup()

    with pytest.raises(ValueError):
        toaster("brioche")
```

### New functionality (even for proof-of-concept)

While it might be tempting to exclude new prototyping code from coverage, I
think it's a bad idea because that's one more thing to remember and track.
Moreover, once a feature is shipped, we usually move on to the next feature and
never clean up our tests. So, it's safer to get into the habit of covering all
new code.

## How to get to 100%

1. Start tracking and displaying the test coverage. Make it easy to find
   uncovered lines (for instance with an HTML report). Have an open discussion
   with your team.
2. Ensure all new code is fully covered, and that coverage can only strictly
   increase. Some Github bots let you enforce this.
3. Before writing new code, check the coverage of the existing functionality,
   and bring it to 100%. Try to structure your tests so that running a single
   test file (e.g. `test_toaster.py`) will cover 100% of its associated code
   file (`toaster.py`). It allows for quicker iterations.
4. Once you're at 100%, use your preferred tool's configuration to fail the
   tests if coverage is not 100%.

```text
Coverage HTML written to dir htmlcov

Required test coverage of 100% reached. Total coverage: 100.00%
```

## Answer to common objections

### 100% test coverage does not mean you're testing everything right.

Absolutely - this point is explicitly stated in this article. I even give an
example situation showing how just checking the test coverage leads to missing
an important test.

100% test coverage does not make your code invulnerable, and it should
evidently not be your only metric. This article is only about the test coverage
metric.

### A test suite that covers 80% is pretty good

Absolutely. It is a good number and a good goal for any
codebase.

However, what about that remaining 20 %? Why are they not tested? Will it be clear
in 2 months why they were not tested? In 6 months? In a year? While it may make
perfect sense not to test them, you should be explicit about that decision and
keep the reason in the code.

If you don't keep the test coverage _metric_ at 100%, then you leave it up to the
code reviewer to challenge your test coverage assumption.

### 100% is a blanket rule that leaves no room for negotiation

Once again, the goal is not to cover 100% of the lines of code - it would be
almost impossible. Thanks to `no cover` markers, you can still decide to
exclude code from coverage. It actually makes this negotiation explicit in the
code, as opposed to implicit during the code review phase coverage.

Consider the example below:

```python
def toast(bread: str) -> str:
    if toast == "brioche":
        raise ValueError("...")
```

Let's say you're fixing a bug in this code, and you find out that the `if`
branch is not covered. You're left to wonder why. The developer did not have
enough time? They decided it was too trivial to need a test?

With an explicit marker, the intent is clear and gives you insight with what is
considered appropriate coverage for _this_ codebase:

```python
def toast(bread: str) -> str:
    if toast == "brioche":  # pragma: no cover, trivial error case
        raise ValueError("...")
```

### It is not feasible in an old codebase

Right, that's probably the case. However, this is not different from proper
testing, monitoring, logging practices. Decide if it's important, then start
with something attainable that brings you closer to the goal, and iterate.

Also, if it is not a desirable goal for the codebase, then for sure don't monitor
test coverage!

### Enforcing 100% test coverage leads to bad tests

It bears repeating: this is not about testing 100% of the lines; this is about
keeping the code coverage metric at 100%. I am not sure how that would lead to
bad tests.

Putting too much focus on one particular metric might lead to gaming behavior.
That does not mean that no metric you should be enforced. I believe that there
isn't enough material and training about what it takes to write good tests.
This is beneficial regardless of whether you enforce test coverage. Also, the
more you talk about test coverage, the more you iterate and learn about
what it takes to write good tests in your language and codebase.

What's sure is that it is straightforward to write bad tests. It takes a lot of
skills, experience, and hygiene to write great, maintainable tests. That's a
topic for another article.

### 100% coverage only matters for languages without a type checker

No. Sure, a type checker might catch some classes of bugs that would require a
test in a language without type checking, but you still need to test
correctness (among others).

### You're creating blind spots

Unless you're reviewing your test coverage report after every single commit,
leaving explicit markers in the code and keeping the metric at 100% is actually
a much safer practice.

When working in code that has been excluded, you'll immediately see the `no cover` marker, perhaps with a comment explaining why the code is excluded. This lets you reconsider the coverage decision.

Any regression in coverage will break the tests.

### You should not exclude code from coverage because of setup cost

This article is not about end-to-end vs. unit testing. I have provided some
examples of code that I sometimes exclude from testing, but your mileage may
vary.

## Changelog

- 09/07/2019: added answer to common objections
- 09/06/2019: added summary, clarified that not testing complicated should only
  apply to extreme cases and a few other points.


All lines of code should be covered by a test or explicitly excluded from test coverage.

Test coverage only matters if it's at 100%


How did Uber go from a 450,000 lines monolithic Python application to more than 1,000 microservices? This short presentation focuses on the technical aspects of this 5-year migration, and concludes with its cultural and management challenges.

I did this talk at the [DevFest](https://devfest.gdgnantes.com/) 2017 in Nantes, France.

<iframe width="560" height="315" src="https://www.youtube.com/embed/O25xbCq_LFU?rel=0" frameborder="0" allowfullscreen></iframe>

<script async class="speakerdeck-embed" data-id="61e701c03a7440b680367ef80c6c5ac5" data-ratio="1.77777777777778" src="//speakerdeck.com/assets/embed.js"></script>

Feel free to also checkout my [professional-programming repo on Github](https://github.com/charlax/professional-programming) which compiles a list of useful resources for programmers.


How did Uber go from a 450,000 lines monolithic Python application to more than 1,000 microservices?

The great microservices migration


It’s always interesting to apply ideas, principles and notions from one field to another. Especially when those two fields are very different.

For instance, software and product design look very different on the surface. While the former is supposedly about pure mathematical rationality, the latter is, at least apparently, about emotions, psychology i.e. about fathoming the messiness of the human mind and the physical world.

Is that true? I don’t think so. Code is written primarily for human to read [1]. Code also usually fixes problems that are coming from the physical world, and thus inherits some of its irrationality.

So, what happens if we apply one of the most regarded designers’ principles to code? Namely, how can Dieter Rams’ [ten principles for good design](https://en.wikipedia.org/wiki/Dieter_Rams) [2] enable us to write better code?

<img src="http://40.media.tumblr.com/9022d63c7546866dc64aaa05b35275ae/tumblr_nhjmijf3EF1qjfk8po1_1280.jpg" width="70%" />

## Good code is innovative

> The possibilities for progression are not, by any means, exhausted.
> Technological development is always offering new opportunities for original
> designs. But imaginative design always develops in tandem with improving
> technology, and can never be an end in itself.

Some problems our code is solving are very old (e.g. moving people across a city). How can we use software, and the machine's capabilities, to better solve them? Even when we’re already using computers (e.g. to scan signed paperwork), how can we use them to go one step further (e.g. ask for an electronic signature), delivering a better experience to the end user?

Even if you have a well established product that is deemed very innovative today, you’re still at the mercy of a new competitor out-innovating you. The tech sector is full of such examples. Companies that stay successful are the ones which don’t fear to constantly disrupt themselves through technological innovation.

## Good code makes a product useful

> A product is bought to be used. It has to satisfy not only functional, but also
> psychological and aesthetic criteria. Good design emphasizes the usefulness of
> a product whilst disregarding anything that could detract from it.

Code is written to be used. As I wrote in [We should not ship code](/posts/2013-08-16-we-should-not-ship-code), we don’t code for the sake of coding. We code because that’s how a lot of problems can be solved.

Because code is an intellectual product, its usability needs to be even more an area of focus. Front-end interfaces and back-end API need to provide good affordance, i.e. their capabilities needs to be self-evident for the user.

Using the pareto principle can be useful here: the codebase should stay focused on the problem at stake, with more than 80% of its LOC being about the business problem. Otherwise, those other functionalities need to be put in libraries, or other services.

## Good code is aesthetic

> The aesthetic quality of a product is integral to its usefulness because
> products are used every day and have an effect on people and their well-being.
> Only well-executed objects can be beautiful.

Note that while this obviously applies to the web interface or the mobile app’s interface, it also applies to a back-end API.

Since code is written to be primarily read by human beings, there’s no reason it can’t take inspiration from literature. Describing how code can be made to be aesthetically pleasing would take a much longer blog post, but here’s some ideas:

- Consistency of code style
- Structure showing clarity of thoughts
- Absence of cruft (`TODO`, `FIXME`, etc.)
- Orthography and grammar for comments
- Appropriate naming
- Conciseness of code, comments, naming, function
- Reusability of concepts and routines

## Good code makes a product understandable

> It clarifies the product’s structure. Better still, it can make the product
> clearly express its function by making use of the user's intuition. At best, it
> is self-explanatory.

Code, more than anything, has a tendency to drift away from the business’ reality. This is one kind of technical debt. When the abstractions that are used in code don’t map the physical world’s reality, they need to be refactored.

One good way to identify when code is impeding a product’s understandability is to document it. If you find yourself having to give too much context about how code and reality differ, then it means code can be improved.

Documentation has a tendency to become out of date, so if you need to invest some time, it’s probably safer to invest it in refactoring the code (taking the opportunity of improving the product as well, refactoring for the sake of refactoring rarely brings enough value) to make it more self-documenting.

## Good code is unobtrusive

> Products fulfilling a purpose are like tools. They are neither decorative
> objects nor works of art. Their design should therefore be both neutral and
> restrained, to leave room for the user’s self-expression.

Code can go too far. Overusing abstraction (here’s an interesting article about how [Go is an anti-abstraction language](http://jeremymikkola.com/posts/2015_08_28_warming_up_to_go.html)), using fancy data structures, overly complex libraries (for instance, using an ORM where it’s not needed), reinventing the wheel: all of those get in the way of having code that is actually maintainable.

This is another point I’ve made in [Maslow’s pyramid of code review](/posts/2015-02-18-maslows-pyramid-of-code-review). Code that is elegant for the sake of being elegant and at the expense of being correct is just getting in the way, and should be refactored or removed.

## Good code is honest

> It does not make a product more innovative, powerful or valuable than it really
> is. It does not attempt to manipulate the consumer with promises that cannot be
> kept.

Over-engineering a solution is a very powerful temptation. Even when a company is moving really fast, and seems to be very far from over-engineering, it can still invest in the wrong area: invest in the wrong tooling too soon, build something from scratch when there are some better off-the-shelf alternatives, etc.

## Good code is lasting

> It avoids being fashionable and therefore never appears antiquated. Unlike
> fashionable design, it lasts many years – even in today’s throwaway society.

More than in any other domain, software development is plagued with short-lived programming libraries, frameworks, patterns. Pick solutions that are battle-tested, and be conservative in your technical choices.

Write tests so that functionality can’t be removed and changed without the future maintainer getting explicit feedback about it.

## Good code is thorough down to the last detail

> Nothing must be arbitrary or left to chance. Care and accuracy in the design
> process show respect towards the user.

Edge cases needs to be accounted for, without going into over-engineering. While it’s fine to sacrifice certain use cases when building an MVP, it’s not what makes the end product delightful. When code elegantly accounts for all edge cases (for instance, through using patterns such as [strategy](https://en.wikipedia.org/wiki/Strategy_pattern)), maintenance is easy and edge cases can be fixed as they come, leading to an amazing user experience.

Also, code is not enough. There’s a lot of context around code that needs to be there:

- Documentation
- Monitoring (operational and business monitoring)
- Alerting
- Testing (unit, integration, end-to-end, capacity)
- Logging and ability to be introspected and debugged

## Good code is environmentally-friendly

> Design makes an important contribution to the preservation of the environment. > It conserves resources and minimizes physical and visual pollution throughout
> the lifecycle of the product.

Code consumes electricity, which is finite resource. When code is made more performant, it not only has an impact on the customer, but also on the environment.

Good code also makes efficient use of data structure and algorithm, and promote reusability. Too often the same piece of functionality is copy pasted between codebases, leading to huge loss in developer efficiency. They should be put in libraries, and open sourced when possible.

## Good code is as little code as possible

<img src="https://upload.wikimedia.org/wikipedia/commons/d/dc/606-Universal-Shelving-System.jpg" width="70%" />

> Less, but better – because it concentrates on the essential aspects, and the
> products are not burdened with non-essentials.
> Back to purity, back to simplicity.

This is my preferred principle. I already talked about it in [We should not ship code](/posts/2013-08-16-we-should-not-ship-code), but the basic idea is that software engineering is not primarily about writing code. It’s about solving problems. It just so happens that a lot of problems can be solved through code - but that does not mean that code solves everything.

How do we write less code?

- By finding existing, off-the-shelf solution and avoiding the NIH (not invented here) syndrome.
- By focusing on the problem at stake, escaping the YAGNI (you ain’t gonna need it) syndrome.
- By spending enough time thinking about the problem, without writing code, so as to design the simplest solution possible (but not any simpler).

Why having less code is better in the long term is pretty obvious: less maintenance, (usually) more performance, less cognitive load for new developers, etc.

## Conclusion

Because code is as much a literary experience as it is a rational undertaking, using principles coming from product design can be useful to think and talk about code.

References:

- [1]: Harold Abelson, Structure and Interpretation of Computer Programs
- [2]: Wikipedia contributors, "Dieter Rams," Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/w/index.php?title=Dieter_Rams&oldid=682540168 (accessed October 1, 2015).
- [3]: Vitsoe at en.wikipedia [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], from Wikimedia Commons


Taking inspiration from Dieter Rams' ten principles for good design, here are the ten principles of good code.

10 principles for good code


As in Maslow's pyramid, each layer requires the previous one. It is useless for code that is charging the wrong customer to be readable.

Code should be:

- **Correct**: does the code do what it's supposed to? Does it handle edge cases? Is it adequately tested to make sure that it stays correct even when other engineers modify it? Is it performant enough for this use case?
- **Secure**: does the code have vulnerabilities? Is the data stored safely? Is personal identification information (PII) handled correctly? Could the code be used to induce a DOS? Is input validation comprehensive enough?
- **Readable**: is the code easy to read and comprehend? Does it make clear what the business requirements are (code is written to be read by a human, not by a computer)? Are tests concise enough? Are variables, functions and classes named appropriately? Do the domain models cleanly map the real world to reduce cognitive load? Does it use consistent coding convention?
- **Elegant**: does the code leverage well-known patterns? Does it achieve what it needs to do without sacrificing simplicity and conciseness? Would you be excited to work in this code? Would you be proud of this code?
- **Altruist**: does the code leave the codebase better than what it was? Does it inspire other engineers to improve their code as well? Is it cleaning up unused code, improving documentation, introducing better patterns through small-scale refactoring?

![](https://36.media.tumblr.com/72e7200921159a4374b7fc163fe0f6f2/tumblr_njwlh7rZui1qgj0nao1_400.png)

_Update Feb 18th_: there's some great comments on [reddit](https://www.reddit.com/r/programming/comments/2wau2x/maslows_pyramid_of_code_review/) about this. I removed the notion of large refactor because I think small refactors have a higher probability of success. I also added a comment about why layering matters.


Maslow's pyramid can help us think about what great code reviews look like.

Maslow's pyramid of code review


> To win a race, the swiftness of a dart
>
> Availeth not without a timely start.

[The Hare and the Tortoise](<http://en.wikisource.org/wiki/The_Hare_and_the_Tortoise_(de_La_Fontaine)>)

Etsy's engineering blog "[Code as Craft](http://codeascraft.com/about/)" can
easily be considered one of the best ones in the industry.

In their definition of "craftsmanship", they make this long quote from [The Pragmatic
Programmer](http://www.amazon.com/Pragmatic-Programmer-Journeyman-Master/dp/020161622X):

> The construction of software should be an engineering discipline. However, this doesn’t preclude individual craftsmanship. Think about the large cathedrals built in Europe during the Middle Ages. Each took thousands of person-years of effort, spread over many decades. Lessons learned were passed down to the next set of builders, who advanced the state of structural engineering with their accomplishments. But the carpenters, stonecutters, carvers, and glass workers were all craftspeople, interpreting the engineering requirements to produce a whole that transcended the purely mechanical side of the construction. It was their belief in their individual contributions that sustained the projects: We who cut mere stones must always be envisioning cathedrals. (Quarry worker’s creed).

There are two parts about craftsmanship: one that is intrinsic, and one that is
extrinsic.

The extrinsic part is perhaps the most obvious one: craftspeople **think
holistically** about what they're building. In some sense, what they aim to
accomplish is much bigger than their work. A great woodworker experiences
empathy about her customer and aspires to build something that will delight the
end-user. They not only support a use case, but go much further and let the
produced artifact transcends it. Well-crafted code not only fixes
a problem, it fixes it elegantly. It does not show any rough patch and
feels a natural solution. The consequence of that is when a problem is not
well-defined, or when the timeline set to fix it is not feasible, engineers
should push back - or at least openly share their concerns, just like a good woodworker would
do. Nobody would ever ask a woodworker to take shortcuts when he's building
the house's framework.

The intrinsic part is about **taking pride in one's work and reflecting about
it**. It's about sharpening one's tools and adding new ones. A great
woodworker would talk for hours about his tools. It's highly probable that his
workshop will be tidy, following the 5S methodology (more about that below).
There's no reason software engineers should feel differently. On the contrary,
they should always be learning, and a great way to learn is to strive for code
quality. When you optimize for speed, you usually don't take time to learn
about your tools and end up misusing them. In the long term, this strategy will
ultimately backfire.

Engineers who take the time to learn about tools, who take time to write
testable & quality code, who take time to carefully think about the
architecture of a service or of an implementation (e.g. following design
patterns) will perhaps be a bit slower in the short term, but this will pay out
ten times in the long term (in quality but also in speed). Striving for quality
in every project, in every task, even mundane (e.g. by automating a test that
could totally be done manually) will enable you to reach the same amount of
quality much faster. Always caring about quality enables continuous
improvement: it might take one hour to learn about how to properly use
a complex but powerful feature of a Python library, but the next time you'll
have 1- probably a much more reliable and fast solution 2- trained your brain
to be constantly learning. Our industry is moving so fast that this is in my
opinion the only durable strategy.

Craftsmanship is ultimately about the worker as a person. Craftspeople are
those who learn and have fun doing it!

![](http://media.tumblr.com/e9919eb6a23489be91c11635608d3763/tumblr_inline_mtcwn0g54e1qfkyvc.jpg)

Here are the [5S](<http://en.wikipedia.org/wiki/5S_(methodology)>) (seiri,
seiton, seiso, seiketsu, and shitsuke) applied to software engineering:

- Sorting: remove code that is never run.
- Set in order: streamline deployment and management through scripts and
  automation.
- Systematic cleaning: remove all TODOs, FIXMEs and temporary code.
- Standardising: remove hacks by improving patterns, abstractions and
  architecture.
- Sustaining: use continuous integration tools to prevent backsliding.

(there's a similar take about 5S applied to software development in [this
article](http://blogs.versionone.com/agile_management/2013/07/25/5s-applied-to-agile-software-development/))

_Pictures from RaidersLight (CC BY-NC-SA) and Alan Cleaver (CC BY)_


Let's take pride in our software engineering craft.

Code as Craft


I'm often asked about resources to learn programming. Here are a few of them.

- [charlax/professional-programming](https://github.com/charlax/professional-programming): a collection of full-stack resources for programmers
- [Coursera](https://www.coursera.org/) (the best IMO)
- [Udacity](https://www.udacity.com/)
- [Udemy](https://www.udemy.com/)
- [Khan Academy](https://www.khanacademy.org/)
- [Learn Python the Hard Way](http://learnpythonthehardway.org/)
- [CodeAcademy](http://www.codecademy.com/)
- [iTunesU](http://www.apple.com/education/itunes-u/) has some good courses.

There are many of them, and this does not even count the books. There are also numerous articles about the best resources to learn coding, and the best method of learning.

Here's my only advice: **nothing beats learning by doing**. Try to build something meaningful and exciting to apply what you learn.


A list of resources to learn programming.

Resources to learn coding


It is easy to be mistaken into thinking that a software developer's job is to ship code.

It is not.

An engineer's job is to solve conceptual problems.

It so happens that out of all the tools humans created, computers are best at solving those, because software is great at dealing with abstractions. However, we will never get recognition from the business if we just ship code. We get recognition if we solve problems and if we do it fast.

That is why the best companies focus on problem-solving in their interviews, rather than _only_ fundamental knowledge. A quick learner can learn whatever she needs to solve things. We are in the learning business. We are always learning new programming languages, databases, libraries, services, patterns... to be faster at solving problems. We also need to learn about the problems itself, and that is why waterfall is rarely a good thing: how can developers fix business problems if they cannot build their understanding of it?

Yesterday, during our show & tell session one developer demonstrated a remarkable tool that would fix one of our long-standing problems. This tool requires minimal integration. A problem was solved without code.

We are in the business of solving intellectual problems. The business is never going to measure our ROI with the number of lines of code we write. The only thing it is concerned with is whether or not the problem was solved efficiently.

It does not mean that everything has to be simple, nor this is a good excuse for fast and loose solutions (ultimately those will make the code less maintainable, and it will be very costly to solve new problems, as requirements tend to grow over time). We should take pride in elegantly solving complex problems. Some complex problems might require a complex solution. That is fine. Just like woodworkers prides themselves in their mastery of their tools, we should be proud of how much we master our tools.

However, our skills are not the end goal. The solution is.


An engineer's job is not to write code - it is to solve business problems.

Code is not the end


## Dependencies

`graphviz` is required for the call graph.

```bash
$ brew install qcachegrind graphviz
$ pip install pyprof2calltree
```

## Profiling

Profile your code with `cProfile` or `Profile`. `pyprof2calltree` won't work with `hotshot`. Store the profile data in a `.pstats` file.

```bash
$ python -m cProfile -o profile.pstats yourscript.py
```

Then run:

```bash
$ pyprof2calltree -i profile.pstats -o profile.kgrind
```

Then open this `kgrind` file with `qcachegrind`:

```bash
$ qcachegrind profile.kgrind
```

![](http://media.tumblr.com/8faf28c1b1d9227330b4f185c48982cd/tumblr_inline_mspsujDXvH1qz4rgp.png)


How to visualize your code's profile with qcachegrind.

Using qcachegrind to visualize Python profiling data


Disclaimer: this article only conveys my personal point of view. Please excuse my English!

## Introduction

I tried numerous text editors: Komodo, Emacs, Nano, Textwrangler, gedit, SPE, [Sublime](http://www.sublimetext.com/), Coda, [Vico](http://www.vicoapp.com/)...

When I discovered TextMate, I really loved it because it was both flexible and
Mac-friendly. Even today, I think that it's a great tool that can greatly
improve productivity.

Yesterday, MacroMates [released](http://blog.macromates.com/2011/textmate-2-0-alpha/) the first public
Textmate 2.0 alpha version. Its authors acknowledged the long wait that preceded
this release by stating that it would "provide some relief to those who have been questioning TextMate’s future".

I began using Vim two weeks ago, and right now I don't see any reason
why I would get back to TextMate. I'll try to explain in details how I got
to use Vim and what are its (numerous) advantages.

## Vim: [Je t'aime moi non plus](http://en.wikipedia.org/wiki/Je_t'aime..._moi_non_plus)

Maybe some other hackers will recognize themselves in this
behavior: even though I've been using Vim for a long time, **I never
took the time to really learn it**. I used to think that Vim was too old to be
interesting: the modal behavior looked really old-fashioned and
counter-intuitive, to say the least.

What's paradoxical is that I kept on using Vim, primarily because it's on all
computers and it's really efficient when modifying system files. Even when
I tried Emacs, I kept on using Vim because it felt so much faster!

With the benefit of hindsight, it was really strange because for all the other
tools I was always trying to improve my productivity (learning keyboard
shortcuts, tuning configuration files, looking for ways to avoid repetition)
but I never took the time to really learn Vim: for instance, two weeks ago
I learned that entering `a` in normal mode would go into insert mode _after_
the cursor…

Why did I suddenly became aware of this paradox?

- I read numerous articles about Vim on Hacker News (I guess it's **useful
  procrastination**).
- TextMate was missing key features like split windows.
- I felt that Vim offered so much flexibility and modularity that it
  would be great to test it.

## Vim's numerous advantages

**Vim is free and open source.** Well-known open source projects are
superior in the long term because they are not dependent on a company's
objectives.

**Vim is superior for text manipulation.** Most of our
work in an editor consists in text manipulation. Contrary to Emacs'
painful shortcuts (from my point of view), Vim's modal mode provides an
_elegant_ and efficient interface. What is more, most of the shortcuts are
easily memorable. TextMate does not provide such a great combination of
ease of use and effectiveness.

**Vim is empowering.** As Bram Moolenaar (Vim's main author) [puts
it](http://www.moolenaar.net/habits.html), "if you spend a lot of time
typing plain text, writing programs or HTML, you can save much of that
time by using a good editor and using it effectively." Vim is great
because it really puts you in charge. The learning curve is quite steep
but it's what's exciting!

**Vim has a vibrant community.** I agree that it's also the case for
Emacs and Textmate. For those who still thinks Vim is old, have a look
at all those repositories on Github!

**Vim's scripting language is straightfoward.** Again, this is really
personal, but I find Lisp not very well adapted to scripting. I really
like Vim's streamlined scripting and configuration files. And you can
also use Python or Ruby.

**Vim's vision is compelling.** As Jeff Bezos says, "maintain a firm grasp of
the obvious at all times." To me it is common sense that a text editor should
focus on efficiency and text manipulation, as Brian Moolenaar puts it:

> It has always been the spirit of Unix to have separate programs that
> do their job well, and work together to perform a bigger task.
> Unfortunately, most editors don't work too well together with other
> programs - you can't replace the e-mail editor in Netscape with
> another one, for example. You end up using a crippled editor. Another
> tendency is to include all kinds of functionality inside the editor;
> Emacs is a good example of where this can end up. (Some call it an
> operating system that can also be used to edit text.)

Vim follows the Unix philosophy, which makes it extremely easy to use it
with other programs. It is not the case for Textmate.

**An editor's configuration files should be put under version control.**
A text editor is arguably the single most important tool for
a developer. This is why it should be possible to constantly improve one's
productivity. TextMate configuration files are not as easy to tweak.

**Vim is everywhere.** Vim is great for sysadmin because it's present on
most computers.

## Stuff I don't like in Vim

- We are in 2011, the user should not have to download syntax files and activate them.
- It should use Perl's regex syntax by default.
- Help files are difficult to follow at times.

More generally, defaults could be improved. For instance, plugins management
could be more automated, instead of having to install Pathogen and using `git submodules`. This could happen outside Vim, with a kind of universal repository
for plugins and a text file that would list all the plugins (like
`.gitmodules`). Plugin would be automatically updated regularly.

Here is another idea to have great defaults: why analyzing all the `vimrc` that
have been posted on the web? This methodology has its flaws, but we would get
a sense of what are the best defaults.

## How to try Vim

If you want to try Vim, just follow these two steps:

- Do **not** use any plugin or anyone else's vimrc. [Really](http://mislav.uniqpath.com/2011/12/vim-revisited/).
- Use `vimtutor` to learn the basics.

## Other articles

- [Checkout my vim configuration](https://github.com/charlax/dotvim) (now in my [dotfiles](https://github.com/charlax/dotfiles))
- [Seven habits of effective text editing](http://www.moolenaar.net/habits.html)
- [Everyone Who Tried to Convince Me to use Vim was Wrong](http://yehudakatz.com/2010/07/29/everyone-who-tried-to-convince-me-to-use-vim-was-wrong/)
- [Vim: revisited](http://mislav.uniqpath.com/2011/12/vim-revisited/): a must-read.
- [Vimcasts](http://vimcasts.org/)

![vim screenshot](http://media.tumblr.com/tumblr_lw7fsqf9ge1qfkyvc.png)


Why taking the time to learn Vim is a smart investment.

Why I keep on using Vim instead of going back to TextMate


These articles were written in 2000, but are still relevant today:

- [Painless Functional Specifications - Part 1: Why Bother?](http://www.joelonsoftware.com/articles/fog0000000036.html)
- [Painless Functional Specifications - Part 2: What's a Spec?](http://www.joelonsoftware.com/articles/fog0000000035.html)
- [Painless Functional Specifications - Part 3: But... How?](http://www.joelonsoftware.com/articles/fog0000000034.html)
- [Painless Functional Specifications - Part 4: Tips](http://www.joelonsoftware.com/articles/fog0000000033.html) (if you have time for only one article, read this one)

Key excerpts:

## Part 1

> So that's giant reason number one to write a spec. Giant reason number two is
> to save time communicating. When you write a spec, you only have to
> communicate how the program is supposed to work once. Everybody on the team
> can just read the spec. The QA people read it so that they know how the
> program is supposed to work and they know what to test for. The marketing
> people use it to write their vague vaporware white papers to throw up on the
> web site about products that haven't been created yet. The business
> development people misread it to spin weird fantasies about how the product
> will cure baldness and warts and stuff, but it gets investors, so that's OK.
> The developers read it so that they know what code to write. The customers
> read it to make sure the developers are building a product that they would
> want to pay for. The technical writers read it and write a nice manual (that
> gets lost or thrown away, but that's a different story). The managers read it
> so that they can look like they know what's going on in management meetings.
> And so on.

## Part 2

> When you're building a product with a team, everybody tends to have their
> favorite, real or imagined pet features that they just can't live without. If
> you do them all, it will take infinite time and cost too much money. You have
> to start culling features right away, and the best way to do this is with a
> "nongoals" section of the spec. Things we are just not going to do. A nongoal
> might be a feature you won't have ("no telepathic user interface!") or it
> might be something more general ("We don't care about performance in this
> release. The product can be slow, as long as it works. If we have time in
> version 2, we'll optimize the slow bits.") These nongoals are likely to cause
> some debate, but it's important to get it out in the open as soon as possible.
> "Not gonna do it!" as George Sr. puts it.

## Part 3

> Program managers are invaluable. If you've ever complained about how
> programmers are more concerned with technical elegance than with
> marketability, you need a program manager. If you've ever complained
> about how people who can write good code never do a good job of writing
> good English, you need a program manager. If you've ever complained about
> how your product seems to drift without any clear direction, you need a
> program manager.

## Part 4

> So. Specs are good, but not if nobody reads them. As a spec-writer, you
> have to trick people into reading your stuff, and you should also probably
> make an effort not to cause any already-too-small brains to leak out
> through eye-sockets.


Some quotable quotes about functional specs.

Category: code