32
loading...
Python's Type Annotations: The Best Of Both Types Worlds
Python is a pretty powerful language with a lot of controversial features. One of them is Typing System, I'm one of the groupies so today my mission will be to convince you it's at least not bad as people claim.
Today, we'll talk about type systems and where Python fits in.
Also, you'll see a way to reduce unwanted bugs and boost your codebase with the Type Annotations feature that was introduced in Python 3.5 (PEP484).
Today, we'll talk about type systems and where Python fits in.
Also, you'll see a way to reduce unwanted bugs and boost your codebase with the Type Annotations feature that was introduced in Python 3.5 (PEP484).
PEP stands for Python Enhancement Proposals, all proposals and specifications of Python's features. It's like ECMA's documents for Javascript.
This feature truly makes Python a better dynamic-typed language.
How many times have you met the following error:
AttributeError: 'X' object has no attribute 'Y'This is one of the most annoying and time-wasting bugs that Python introduce. There are more errors that occur because of non-safety with variable types and non-existing type validations (or tests in general).
As a dynamic-typed language, all the type errors will appear in runtime.
Try to recall how frustrating it meeting this type of exception in production.
For those of you who know Javascript, you can see the huge hype over Typescript in the last few years. Why is that? Fewer bugs!
Nowadays people understand that bugs in production are way worst from adding types in their code.
As a dynamic-typed language, all the type errors will appear in runtime.
Try to recall how frustrating it meeting this type of exception in production.
For those of you who know Javascript, you can see the huge hype over Typescript in the last few years. Why is that? Fewer bugs!
Nowadays people understand that bugs in production are way worst from adding types in their code.
But wait, let's go back and explain the fundamentals about type systems, and why I say stuff.
You arrived at the airport for taking your flight to the annual vacation.
In the entry of the airport a security guard approaches to you and ask:
In the entry of the airport a security guard approaches to you and ask:
Big Security Guy: "Let's say you have a choice between 2 options.
(1) Passing a security check which includes long lines and cursings.
(2) you go freely into your airplane and during the flight, you'll be checked.
What would you prefer?"
This story looks exactly like the war between dynamic and static languages.
Your code types will be checked in the compilation process.
Those types of languages main pros:
Those types of languages main pros:
If you know about a bomb in your airport, you rather choose this choice (1).
e.g: Java, C, C++Your code types will be checked only during runtime.
Those types of languages main pros:
Those types of languages main pros:
The problem will be when there is a bomb on your flight.
For more safe flight (/run) your codebase must have tests and validations that will prevent issues in runtime.
For more safe flight (/run) your codebase must have tests and validations that will prevent issues in runtime.
e.g: Python, Javascript, R, JuliaPython's Place
Python is a pure dynamic-typed language.
Our goal will be to improve the downsides of dynamic-typed languages.
Python is a pure dynamic-typed language.
Our goal will be to improve the downsides of dynamic-typed languages.
We can agree that both types of languages codebases should have tests, but we live in an imperfect world, we don't have always the time to perform that tests, especially in large codebases in which high coverage is hard to target.
Dynamic languages, as we said, are less safe in their essence.
There are many cases that types are difficult to track. A common example is that it very hard tracking types of arguments in deep nested function calls.
So to revealing the bugs that occur because of programmer's mistakes, tests should be written.
How can we increase our safety without making any efforts and spending more time?
Dynamic languages, as we said, are less safe in their essence.
There are many cases that types are difficult to track. A common example is that it very hard tracking types of arguments in deep nested function calls.
So to revealing the bugs that occur because of programmer's mistakes, tests should be written.
How can we increase our safety without making any efforts and spending more time?
Type Annotations are simply type-hints that can be attached to variables and functions declarations.
They can be used by third-party tools such as type checkers, IDEs, and linters for real-time warnings while coding.
The Python runtime does not enforce function and variable type annotations, so we didn't actually make Python more static.
Also, Python's interpreter will relate the Type Annotations as comments, so no effect on runtime.
We got statically-typed language experience with the dynamic-typed language.
They can be used by third-party tools such as type checkers, IDEs, and linters for real-time warnings while coding.
The Python runtime does not enforce function and variable type annotations, so we didn't actually make Python more static.
Also, Python's interpreter will relate the Type Annotations as comments, so no effect on runtime.
We got statically-typed language experience with the dynamic-typed language.
Let's introduce some central features of Type Annotations.
Builtin Types
It is possible hinting each one of the builtin types:
It is possible hinting each one of the builtin types:
int, str, float, bool, bytes, object, None.Special Types
Notice that in Python3.9+ most of the following types are changed and won't be imported anymore from typing module anymore, e.g, List -> list.
Any Any type is possible.List[str] List of str objects.Tuple[int, int] Tuple of two ints.Tuple[int, ...] Tuple of int objects.Dict[int, int] Dict with int keys to int values.Iterable[int] Iterable which contains int objects.Sequence[bool] Sequence of booleans.By the way: Sequence is an Iterable with defined length and extra functionality.
def greeting(name: str) -> str:
return 'Hello ' + nameIf we will run the following code
greeting(3) we will get the following error:error: Argument 1 to "greeting" has incompatible type "int"; expected "str"It is possible to define aliases to types for more readable code.
Url = str
def retry(url: Url, retry_count: int) -> None:
passTypeVar is a factory that can generate parameterized types. It can be very useful in classes that can hold different data types.
from typing import TypeVar, Generic
T = TypeVar('T')
class Stack(Generic[T]):
def __init__(self) -> None:
# Create an empty list with items of type T
self.items: List[T] = []
def push(self, item: T) -> None:
self.items.append(item)
def pop(self) -> T:
return self.items.pop()
def empty(self) -> bool:
return not self.items
# Construct an empty Stack[int] instance
stack = Stack[int]()
stack.push(2)
stack.pop()
stack.push('x') # Type errorIt is possible to create an object without specifying the type, the first object that will be inserted into the data structure will determine the class parameterized type (type inferring).
Sometimes you will accept several types for your variable.
Union
Specifying a set of possible types, each one of them will be accepted.
Union
Union[T1,...,Tn]Specifying a set of possible types, each one of them will be accepted.
def f(x: Union[int, str]) -> None:
if isinstance(x, int):
x + 1 # OK
else:
x + 'a' # OK
f(1) # OK
f('x') # OK
f(1.1) # ErrorOptional
Optional will indicates the variable holds a specified type or
It actually equals to
Optional[T]Optional will indicates the variable holds a specified type or
None.It actually equals to
Union[T, None].def strlen(s: str) -> Optional[int]:
if not s:
return None # OK
return len(s)For many usefull annotations I recommend exploring mypy docs.
Better than a compiler. You got your potential errors while developing!
Here few examples from Pycharm docs to understanding how awesome this thing is.
In the next example, we can see that we perform an invalid assignment.
In the next example, we can see that we perform an invalid assignment.
Here we use the
Final type to mark a variable as a constant. Assignment to this constant will lead to an error.Here the IDE enforces Enum values for preventing any programmer mistakes, unified format.
Literal type makes it possible to supply specific primitive values for the variable.Here we use TypedDict as a schema for function argument.
Those examples are a drop in the ocean, I hope it's clear now how the IDE enforces bugs and acting as a real-time compiler enforcer.
From my experience, refactoring with type validations can save a lot of time and unwanted bugs in production.
During refactoring, you usually change parts of code that can affect many other places. As we said, for catching unwanted errors in many places we need good tests coverage, without it, there is a good potential for programmer mistakes that won't be detected.
So, if your coverage is not good as you think, type annotations will help.
During refactoring, you usually change parts of code that can affect many other places. As we said, for catching unwanted errors in many places we need good tests coverage, without it, there is a good potential for programmer mistakes that won't be detected.
So, if your coverage is not good as you think, type annotations will help.
Docstrings are usually the solution for documentation in Python, there are many different formats such as Epytext, reST, Google (my favorite), and more.
There is no one standard strict format for docstrings, so it can be pretty difficult to enforce and check type problems in the codebase.
There is no one standard strict format for docstrings, so it can be pretty difficult to enforce and check type problems in the codebase.
*Docstrings for humans, Type Annotations for linters. *
For reaching maximal experience of documentation, use type hints for enforcing any errors, and document only the non-trivial stuff.
For reaching maximal experience of documentation, use type hints for enforcing any errors, and document only the non-trivial stuff.
As we know, sometimes our IDEs are pretty annoying. especially when you have
Type Annotations make suggestions feature more accurate and easier. Typing your variable with a specific type will autocomplete you with these specific type suggestions.
int variable and string suggestions are made. Type Annotations make suggestions feature more accurate and easier. Typing your variable with a specific type will autocomplete you with these specific type suggestions.
Type Annotations are not only a way of enforcing types in your code. Like every feature in Python, it is possible to access the annotations of specific code.
PEP3107 (Function Annotations) specify how can we retrieve metadata from our code, using Type Annotations with
PEP3107 (Function Annotations) specify how can we retrieve metadata from our code, using Type Annotations with
__annotations__ property.def foo(a: 'x', b: 5 + 6, c: list) -> max(2, 9):
...
> foo.__annotations__
{'a': 'x',
'b': 11,
'c': list,
'return': 9}Why is it good?
There are several powerful usages for this feature, a few of them are Database query mapping, Foreign-language bridges, RPC parameters encoding, and Schemas validations.
A really good example of this feature usage is FastAPI.
FastAPI is a fast and modern web framework based on Python Type Annotations. Why is that? Why shouldn't we use Flask and Django and forget about those types? Value.
Additionally, to speed, FastAPI gives few powerful features based on type hints, for example:
OpenAPI Generation
Whenever you run your server an OpenAPI (Swagger) specification file will be generated automatically, you don't have to document your endpoints anymore!
Schemas and Data Models
Another one is data models, Using the module Pydantic, with the simple specifications of your REST requests and arguments, validation and conversion will be performed so you can hold your objects without messing around with Jsons.
There are several powerful usages for this feature, a few of them are Database query mapping, Foreign-language bridges, RPC parameters encoding, and Schemas validations.
A really good example of this feature usage is FastAPI.
FastAPI is a fast and modern web framework based on Python Type Annotations. Why is that? Why shouldn't we use Flask and Django and forget about those types? Value.
Additionally, to speed, FastAPI gives few powerful features based on type hints, for example:
OpenAPI Generation
Whenever you run your server an OpenAPI (Swagger) specification file will be generated automatically, you don't have to document your endpoints anymore!
Schemas and Data Models
Another one is data models, Using the module Pydantic, with the simple specifications of your REST requests and arguments, validation and conversion will be performed so you can hold your objects without messing around with Jsons.
class Item(BaseModel):
name: str
description: Optional[str] = None
price: float
tax: Optional[float] = None
app = FastAPI()
@app.post("/items/")
async def create_item(item: Item):
return itemThis is also a drop in the ocean. As a Backend Engineer who worked with few web frameworks, I can surely say that these features are truly amazing and powerful.
MyPy is a static type check for Python 2 and 3.
We can treat MyPy as our compiler which does not output any executable, only checks and find our bugs in the input codebase.
MyPy can type check your code and prevent bugs, it acts like a linter that runs as a static analyzer.
It is possible to create a configuration file for a more optimized and customized experience.
We can treat MyPy as our compiler which does not output any executable, only checks and find our bugs in the input codebase.
MyPy can type check your code and prevent bugs, it acts like a linter that runs as a static analyzer.
It is possible to create a configuration file for a more optimized and customized experience.
# main.py
def print_hi(name: str) -> None:
print(f'Hi, {name}')
> mypy main.py
> main.py:4: error: Argument 1 to "print_hi" has incompatible type "int"; expected "str"MyPy and IDE that support Type Annotations are a powerful combination.
IDE for real-time warning and MyPy for final checkings (Possible in CI).
IDE for real-time warning and MyPy for final checkings (Possible in CI).
Recalling: My goal is to convince the haters that those type annotations are not bad as they claim.
Type Annotations like every controversial feature have few drawbacks.
I'll talk about the most popular.
Type Annotations like every controversial feature have few drawbacks.
I'll talk about the most popular.
Adapting Type Annotations with the existing codebase is a challenging topic, small repos are not such an issue because adding types will be pretty fast. Large codebases cannot add Type Annotations quickly, it can be challenging.
You can try those two approaches combined for achieving types codebase:
You can try those two approaches combined for achieving types codebase:
MonkeyType and PyAnnotate tools can infer the types of your code during runtime.# type: ignore and move on. Subset by subset, Part by part.
There is a big advantage having a big codebase with Type Annotations, refactoring will be easier and bugs can be detected fast.Well-documented code will have types and documentation already, so what is the cost of moving the types declaration into type annotation?
If you don't document your code, still, insert a word per variable or method declaration doesn't seem very time-consuming.
If you don't document your code, still, insert a word per variable or method declaration doesn't seem very time-consuming.
Agree. In the end, it's a matter of taste, I met people that claim that type annotation contributes to the readability of their code.
Like everything it's a cost versus benefits game. In the beginning, it can be uglier than working without any annotation but I really think that after a while you are getting used to it.
Like everything it's a cost versus benefits game. In the beginning, it can be uglier than working without any annotation but I really think that after a while you are getting used to it.
As PEP484 said:
Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.To conclude, type annotation can contribute a lot to your codebases.
Python rules.
Python rules.
32