DAVID BRANCACCIO: There’s a little sweet moment, I’ve got to say, in a very intense book– your latest– in which you’re heading out the door and your wife says what are you doing? I think you say– I’m getting– I’m going to buy an envelope.

KURT VONNEGUT: Yeah.

DAVID BRANCACCIO: What happens then?

KURT VONNEGUT: Oh, she says well, you’re not a poor man. You know, why don’t you go online and buy a hundred envelopes and put them in the closet? And so I pretend not to hear her. And go out to get an envelope because I’m going to have a hell of a good time in the process of buying one envelope. I meet a lot of people. And, see some great looking babes. And a fire engine goes by. And I give them the thumbs up. And, and ask a woman what kind of dog that is. And, and I don’t know. The moral of the story is, is we’re here on Earth to fart around. And, of course, the computers will do us out of that. And, what the computer people don’t realize, or they don’t care, is we’re dancing animals. You know, we love to move around. And, we’re not supposed to dance at all anymore.

https://crawshaw.io/blog/new-year

Good new music has a way of feeling like the best thing you’ve ever heard, until the novelty wears off. I don’t want to be updating this list every time that happens, so I’ll only revisit it once a year.

Year	Cover Art	Album
1995		Mobb Deep – The Infamous
1996		Fugees – The Score
1997		The Notorious B.I.G. – Life After Death
1998		OutKast – Aquemini
1999		Mos Def – Black On Both Sides
2000		Common – Like Water For Chocolate
2001		Kalamashaka – Ni Wakati
2002		Clipse – Lord Willin'
2003		Gang Starr – The Ownerz
2004		Madvillain (MF DOOM & Madlib) – Madvillainy
2005		Kanye West – Late Registration
2006		Ukoo Flani Mau Mau – Dandora Burning
2007		Kanye West – Graduation
2008		Fleet Foxes – Fleet Foxes
2009		Mos Def – The Ecstatic
2010		Big Boi – Sir Lucious Left Foot: The Son Of Chico Dusty
2011		Kendrick Lamar – Section 80
2012		Kendrick Lamar – Good Kid, M.A.A.D City
2013		Intransit Riddim – Various Artists
2014		J. Cole – 2014 Forest Hills Drive
2015		Kendrick Lamar – To Pimp A Butterfly
2016		Frank Ocean – Blonde
2017		Kendrick Lamar – Damn
2018		Mac Miller – Swimming
2019		Common – Let Love
2020		Armand Hammer – Shrines
2021		Tyler, The Creator – Call Me If You Get Lost
2022		Billy Woods – Aethiopes
2023		Djungle & Kitu Sewer – Kaunenge
2024		Kitu Sewer – Makali Man
2025		Clipse – Let God Sort Em Out

• Using setdefault
• Using defaultdict
• Implementing __missing__

While the advice there still stands, I’ve been reading through Fluent Python and came across two things worth being aware of when subclassing dict:

1. __contains__ doesn’t call __missing__, so k in d returns False for keys not yet set, even if __missing__ would handle them
2. dict.get doesn’t call __missing__, so .get(k) won’t use your custom default

Both are by design and often what you want. __missing__ is only used by d[key], not other dict methods. Even defaultdict follows the same rule: only d[key] triggers the default factory and methods like .get() and in do not.

Overriding get is tempting but tricky. A naive implementation might look like this:

class M(dict):
    def __missing__(self, key):
        value = "my default value"
        self[key] = value
        return value

    def get(self, key, default=None):
        try:
            return self[key]  # triggers __missing__ if key is absent
        except KeyError:
            return default    # dead code, never reached

This doesn’t work. self[key] calls __missing__ instead of raising KeyError, so the except block is unreachable and default is effectively useless:

>>> m = M()
>>> m.get("x", "fallback")
'my default value'          # expected "fallback", got __missing__ value instead
>>> m
{'x': 'my default value'}  # side effect: key was set in the dict

There’s no clean way to honour both the __missing__ default and the default parameter in get. It’s better to accept that get and __missing__ serve different purposes and leave get alone. Note that this applies to dict subclasses specifically. If you subclass UserDict instead, get calls __getitem__ internally and __missing__ is triggered. With a dict subclass, get is implemented in C and bypasses __getitem__ altogether. The behaviour is inconsistent across the standard library depending on which base class you use.

As for __contains__, overriding it is a design decision. If your subclass provides a value for every possible key, returning True always can be reasonable:

def __contains__(self, key):
    return True

Only thing to keep in mind in this case is that "x" in m returns True even when m is {}, which looks confusing.

Update 2025-11-25: After trying this for about three weeks, I’ve found that disabling Javascript by default is really inconvenient. Many sites just don’t work properly without it, and having to manually enable it every time is a hassle. I guess that’s the tradeoff one would have to make in this case, but I probably just don’t care enough at the moment to go ahead with this. Of course I might revisit this again later, but for now it’s back to the default browsing experience.

This is something I always knew you could do but never considered doing - until I came across this site. I was browsing some pages on that site, and had to leave to check something else on a different tab. After a while, I noticed something weird with the tabs:

I’m pretty sure I didn’t search for “censorship on hackernews”, or “how to hack coworkers phone”. And I’m certainly not asking ChatGPT for pickup line suggestions 😂. What was going on?

Well, Javascript.

That message is shown when you go back to the tabs. Basically, the site uses Javascript to change the page title when the tab is not in focus.

This is not actually the first time I’ve seen this. The same thing happens on sizeof.cat, but there it’s done just for fun. That’s why I never thought twice about it when I first saw it there. The author of マリウス also acknowledges that he got the idea from the same source:

I got the inspiration for this from this post by sizeof.cat, […].

While sizeof.cat uses this feature purely for the lulz, I believe it can serve as an effective way to encourage people to disable JavaScript in their browsers by default, and to be very selective about which websites they enable it for.

I think the fact that he decided to use this trick for this purpose is genius. Props to him. It’s really effective.

Most of these titles are not too bad (though they would still raise eyebrows), but some would be quite embarrassing if seen in your tabs by others.

How do you explain to someone that the “jeff bezos nudes” tab they saw in the background wasn’t actually a Google search for “jeff bezos nudes”, but a joke from some blog on the internet?

I’m sure there are some security features on browsers to restrict the harm JS can do on your machine - but the fact that something like this can happen behind your back really makes you think.

Of course it’s going to be a pain to turn off JS by default and manually enable it whenever I visit a site I need and find that it just won’t work unless I have it enabled. But I’ll try it out for a while and see how it goes.

Side note: I also learned about Punycode on that site, so - worth the visit.

For example:

id | model     | competitor_type
--------------------------------
1  | Ariya     | NULL
2  | Frontier  | NULL
3  | Rogue     | Competitor
4  | Sentra    | Non-competitor

SELECT * FROM table WHERE LOWER(competitor_type) != "non-competitor";

I expected to see ids 1, 2 and 3 as a result, but only got 3.

This makes sense when you think about it because comparisons can’t be carried out against unavailable values. So if NULL values are expected in the result, they should explicitly be included in the query.

SELECT * FROM table WHERE (LOWER(competitor_type) != "non-competitor" OR competitor_type IS NULL);

This can also be handled by using COALESCE to get values from NULL, or by using the NULL-safe equal <=>, but I think it’s a lot more clear to just check for NULL explicitly.

SELECT
    *
FROM
    table_1
LEFFT JOIN
    table_2
USING (a,b);

At first I thought BigQuery was pretty smart and just assumed I meant LEFT instead. Maybe it’s a common typo and there’s an alias for it. So I got curious and tried out this:

SELECT
    *
FROM
    table_1
LEFFFFTFT JOIN
    table_2
USING (a,b);

No error. That can’t be right.

That’s when it hit me. This query isn’t doing a LEFT JOIN. It’s using the incorrectly spelled name as an alias for table_1 and doing an INNER JOIN instead. Basically this:

SELECT
    *
FROM
    table_1 AS LEFFT
JOIN
    table_2
ON LEFFT.a = table_2.a AND LEFFT.b = table_2.b

So the query works fine but the results would be incorrect.

This is one benefit of syntax highlighting I’ve never really thought about until now. The only reason I noticed this quickly was by seeing LEFFT wasn’t highlighted as I expected. The results of the query looked fine at first glance so this would have definitely bitten me somewhere down the line if it got to production.

A with statement does not create a scope (like if, for and while do not create a scope either).

As a result, Python will analyze the code and see that you made an assignment in the with statement, and thus that will make the variable local (to the real scope).

In Python variables do not need initialization in all code paths: as a programmer, you are responsible to make sure that a variable is assigned before it is used. This can result in shorter code: say for instance you know for sure that a list contains at least one element, then you can assign in a for loop. In Java assignment in a for loop is not considered safe (since it is possible that the body of the loop is never executed).

Initialization before the with scope can be safer in the sense that after the with statement we can safely assume that the variable exists. If on the other hand the variable should be assigned in the with statement, not initializing it before the with statement actually results in an additional check: Python will error if somehow the assignment was skipped in the with statement.

A with statement is only used for context management purposes. It forces (by syntax) that the context you open in the with is closed at the end of the indentation.

source

As far as I know, Spotify doesn’t provide any way to batch delete things, so I created a spotify-cleaner for this.

I used it to delete everything (all liked songs, albums, podcasts, and playlists), but it can also delete just a few selected items.

If this is something you feel might also be useful to you, check the repo for instructions on how to use it.

So I ran man printf instead. Output:

PRINTF(1)                                       User Commands                                       PRINTF(1)

NAME
       printf - format and print data
 ...

Then man 3 printf:

printf(3)                                  Library Functions Manual                                 printf(3)

NAME
       printf,  fprintf,  dprintf, sprintf, snprintf, vprintf, vfprintf, vdprintf, vsprintf, vsnprintf - for‐
       matted output conversion

LIBRARY
       Standard C library (libc, -lc)

SYNOPSIS
       #include <stdio.h>
...

And I noticed what I’d been missing. The first output has User Commands at the top, while the other one has Library Functions Manual.

As it turns out, man pages are divided into sections. I’ve never thought to check man man, but it explains this.

MAN(1)                                        Manual pager utils                                       MAN(1)

NAME
       man - an interface to the system reference manuals

SYNOPSIS
       man [man options] [[section] page ...] ...

...

A manual page consists of several sections.

A section, if provided, will direct man to look only in that section of the manual.
The default action is to search in all of the available sections following a
pre-defined  order (see DEFAULTS), and to show only the first page found,
even if page exists in several sections.

    The table below shows the section numbers of the manual followed by the types of pages they contain.

    1   Executable programs or shell commands
    2   System calls (functions provided by the kernel)
    3   Library calls (functions within program libraries)
    4   Special files (usually found in /dev)
    5   File formats and conventions, e.g. /etc/passwd
    6   Games
    7   Miscellaneous (including macro packages and conventions), e.g. man(7), groff(7), man-pages(7)
    8   System administration commands (usually only for root)
    9   Kernel routines [Non standard]

To check what sections are available for a command:

man -f <command>

# for example:
man -f printf
printf (1)           - format and print data
printf (1p)          - write formatted output
printf (3)           - formatted output conversion
printf (3p)          - print formatted output

You may need to run sudo mandb if you get the error nothing appropriate with the above command.

Anyway, Vim aside, I’ve decided to learn how to type properly. I can type pretty fast, but I only ever use my index and middle fingers. I also have to look at the keys when typing. I think this really limits my speed. And not just my typing speed - the speed of all my work in general. The fact that I have to look down to type, then up to confirm that what I’ve written is correct means I’m using twice the time to write - and I do this a lot in the course of a normal working day.

Something else that really bugs me is when I’ve typed a whole sentence only to look up and realize the window/textbox I was supposed to be typing on is no longer in focus, and all my typing wasn’t doing anything, or worse, was triggering random shortcuts in a different window.

keybr has been awesome for practicing. I also try to practice opportunistically while working, which has made me very slow, but I think it’s a worthwhile investment.

I work remotely for a company in the UK, and we use Slack for communication. The timezone difference is only two/three hours, so the whole team is usually all online at the same time for most of the day. I (and I’m sure most other members of the team) just find it a lot easier to pull someone in for a ‘quick’ huddle each time there’s something to discuss. I always hesitate before doing this, but end up doing it anyway most of the time. The fact that it is very easy to do doesn’t help. I think it does more harm than good.

More than a few times I try to invite someone for a huddle, and they’re not available. I then have to write down whatever I wanted us to discuss - and it always ends up feeling more productive. I think better when I write. Or, maybe writing actually forces me to think about the thing more - something I may have been avoiding and choosing the easier option of ‘discussing’ it instead. I even come up with the solution sometimes in the middle of writing, and find the call, or even the message itself, not to be necessary.

Another benefit of writing things down is that you can actually refer back to the messages months/years down the line when you can’t figure out why a certain decision was made. I know Slack huddles have threads attached to them but we rarely ever use them. Handovers are a lot easier when you can actually search for something in threads and follow discussions that were made in the past.

Writing is also great practice for communication skills as things usually need to be a lot more clear written down than spoken.

I know these kinds of impromptu calls are distracting, but I understand why people do them. It’s just the easiest thing to do in the moment. They’re hardly ever necessary though.

In my experience, I have never been in a situation where I would be completely blocked unless I got into a call to discuss something. I’m never in need of an immediate answer to be able to continue with my work. And even if I am, there’s no guarantee that calling someone is going to get me that answer. They may not be available, and then I’ll still have to write down whatever I wanted anyway.

It’s usually not worth breaking someone’s focus over whatever issue I feel needs immediate attention. It’s not easy, but it’s worth embracing asynchronous communication more.

Write something down as clearly and precisely as you can and send it. Don’t expect an immediate reply. Find something else to do in the meantime.

I think that’s a better way to work.

Update (2024-09-25)

Found this link that I think supplements this post well: No “Hello”, No “Quick Call”, and no Meetings Without an Agenda

The schema looked something like this:

class Schema(BaseModel):
    dates: List[str]
    start_date: Optional[str]
    end_date: Optional[str]
    ...

    @validator("dates")
    def update_date_format(cls, v):
        ...

    @validator("start_date")
    def set_start_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][0]
        return v

    @validator("end_date")
    def set_end_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][-1]
        return v

My assumption in the last two validators was that dates would always be available because it’s a required field. To my surprise, an IndexError was always raised on set_start_date.

So I thought, ok, I’ll raise a ValueError myself on update_date_format since the required field check doesn’t seem to be working.

class Schema(BaseModel):
    dates: List[str]
    start_date: Optional[str]
    end_date: Optional[str]
    ...

    @validator("dates")
    def update_date_format(cls, v):
        if not v:
            raise ValueError("dates is a required field")
        ...

    @validator("start_date")
    def set_start_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][0]
        return v

    @validator("end_date")
    def set_end_date(cls, v, values):
        # some logic here that expects `dates` to exist
        v = values['dates'][-1]
        return v

Again, an IndexError was raised on set_start_date. It’s like my check on update_date_format was completely ignored. I even put a breakpoint on the line to make sure it was being hit. It was.

So, validation errrors aren’t raised immediately. They’re collected and returned all at once in the response.

Despite seeing this countless times in 422 responses, I had never thought about it. It made perfect sense. You don’t want to return one error at a time as an API response. It’s much better to return a response detailing everything that’s wrong with the payload.

From the docs:

One exception will be raised regardless of the number of errors found, that ValidationError will contain information about all the errors and how they happened.

The important thing to remember is that this only happens for validation errors (i.e errors raised through ValueError or AssertionError). This is why the IndexError was always raised immediately it happened.

How to stop validation on the first error

According to this answer:

If you have checks, the failure of which should interrupt the further validation, then put them in the pre=True root validator. Because field validation will not occur if pre=True root validators raise an error.

For example:

class PayloadValidator(BaseModel):
   emailId: List[str]
   role: str

   @root_validator(pre=True)
   def root_validate(cls, values):
       if not values['emailId']:
           raise ValueError("Email list is empty.")
       return values

   @validator("emailId")
   def valid_domains(cls, emailId):
       return emailId

Before I get into the things that really bothered me, here’s what I liked:

1. Online accounts

This is more of a Gnome thing than Ubuntu, but still, I love that I can add all my online accounts in the settings and have everything synced up automatically in the calendar and email apps. I love KDE Plasma, but this is the one thing I really miss in it.

I actually don’t care much for email, but calendar notifications is something I really need. I always find myself forgetting about scheduled meetings. Of course I could find another way to deal with this, but it would be really nice if I could have it OOTB with my distro.

2. Wallpapers

Not really a major thing but Ubuntu and ElementaryOS are the two distros I’ve seen that always have the best wallpapers shipped by default.

3. Somewhat consistent themeing

I think Ubuntu does this really well with the Yaru theme. I say somewhat because this is Gnome, and there are always issues with themeing Gnome.

That’s about it for what I liked.

Here’s what I didn’t like:

1. Installer didn’t work

This is a big problem. Ubuntu released a new installer written in flutter and it just didn’t work for me.

It kept crashing in the middle of the setup process (before I actually got to the final install step), and when I finally got to the final step and clicked install, it froze at 5%.

I tried multiple times and even downloaded an update for the installer. Same thing.

Finally, I decided to go back to the website and download the legacy installer version - which worked fine.

This should have been a huge red flag for me. I should have given up on the whole thing the moment I had issues with the installer. But oh well, I apparently had time to waste.

This was a bad one.

Everything else after this is based on personal preference so not anything really wrong with Ubuntu itself.

2. Huge window title bars

I think the window title bars on Gnome are ridiculously huge. I can’t stand them. I tried getting around that with a theme that offers a “compact” version, but -

3. Themeing is such a pain

Why do I need to install an extra piece of software (Tweaks) to be able to change the default system font? I don’t understand the reasoning behind this. At the very least, this should be an option in the settings.

Fonts aside, applying a new shell theme doesn’t really change the theme for all apps. Something to do with “Libadwaita” and legacy apps. I didn’t look into it enough to figure out what was going on. But apps like the terminal and the file manager just didn’t want to be themed differently. I use these apps a lot so it would be too distracting for me to have them not looking the same as everything else.

4. Different ways to install software

This is totally me having been spoiled by the conveniences of the AUR. When I install an arch based distro, I have a script that installs yay, and uses it to install everything else from wherever it’s available (official repos or AUR).

yay -S literally-anything. It Just Works ™.

With Ubuntu, I have to figure out if the thing is in the repos. I may have to add some new repos, add GPG keys and run system updates. Maybe the thing is a snap. In that case, I need to run some different commands to install snaps.

I wasn’t even able to install previous versions of Python ¹. Something about it being a security issue for non-lts versions (can’t remember the exact error).

There’s probably something out there like yay but for Ubuntu - I don’t know. But as it is, it’s all just too much work.

This answer does a good job explaining how it works.

I’ve not had a need for it yet, but it’s a good thing to keep in mind for when I do end up needing it.

incron is an “inotify cron” system. It consists of a daemon and a table manipulator. You can use it a similar way as the regular cron. The difference is that the inotify cron handles filesystem events rather than time periods.

I use it to auto-commit changes made to specific files to version control.

Example usage

View incrontabs:

incrontab -l

Add/edit incrontabs:

incrontab -e

# commit and push when file is modified and saved
/home/wangonya/dir_to_track	IN_MODIFY	cd /home/wangonya/dir_to_track; git commit -a -m 'autocommit on change'; git push

Archwiki for more info.

It usually goes something like this:

• Start working on problem. Do some research. Try out A.
• A doesn’t work. Do some more research. Try B.
• B doesn’t work.
• Try C. Nothing.
• At this point I’m already tired so I think to myself, “A seems like a pretty good idea. Why didn’t it work? Let’s try it again.”.
• Oh, that’s why A didn’t work. Of course.
• Try some variation of B.
• Try C again.
• Think about A some more…

So much time is wasted, with very little progress made. Even worse, if this is at the end of the day and I have to leave it pending and come back to it the next day - or if I have to stash the changes and work on something else for a while then come back to it later, then the cycle begins all over again.

If notes are taken at each step of the way, it’s clear what works and what doesn’t - and why. This isn’t just helpful in the moment. Chances are I’ll come across a similar issue in the future. I can’t always rely on memory on these things. Experience has shown it to be very unreliable.

As much as I plan to mostly use pen and paper for this, it should also give me ample opportunity to blog more. Many problems should be general enough for someone¹ else out there to benefit from.

This was initially a post about adding what’s playing on Spotify to i3blocks. I had just started using i3blocks then and thought it was pretty cool how easy it was to do. I’ve since discovered the possibilities are endless. You can add pretty much anything that outputs text to the statusbar.

Below are some examples of what’s possible.

Showing and controlling music

I like seeing what’s playing without having to bring up the Spotify app every time. If I can also control playback without bringing up the app, even better.

By default, i3 comes with i3bar. I’m sure I could achieve the same thing on i3bar with enough time and effort, but its just much easier to do it with i3blocks.

Info on what’s playing can be gotten using playerctl or spotify-cli-linux.

Then add the command to the i3blocks config:

[spotify]
command=spotifycli --status
color=#00FFB3
interval=2

I’ve opted to create a separate script file so I can handle mouse clicks.

#!/usr/bin/env bash

# i3blocks spotify script

spotifycli --status

case $BLOCK_BUTTON in
  1) spotifycli --prev ;;  # left click, previous
  2) spotifycli --playpause ;; # middle click, play/pause
  3) spotifycli --next ;; # right click, next
esac

; i3blocks config

[spotify-playback-status]
command=spotifycli --playbackstatus
color=#00FFB3
interval=2
separator=false

[spotify]
command=~/.config/i3blocks/scripts/spotify
color=#00FFB3
interval=2

I also tried this on Strawberry player using playerctl and it worked just as well.

Showing Wakatime stats

Using wakatime-cli:

; i3blocks config

[wakatime]
label=
command=wakatime-cli --today
color=#3daee9
interval=300

mysqldump -u username -ppassword db_name table_name > dump.sql

For multiple tables, separate the table names with a whitespace:

mysqldump -u username -ppassword db_name table_name_1 table_name_2 > dump.sql

To only get tables with a certain prefix, e.g pref_:

mysqldump db_name $(mysql -D db_name -Bse "show tables like 'pref\_%'") > dump.sql
# flags: -B batch, -s silent mode, -e execute

To only dump rows that meet a specific criteria:

mysqldump -u username -ppassword db_name table_name --where="where clause here" > dump.sql

Importing the dump file:

mysql -u username -ppassword db_name < dump.sql

The Feynman algorithm

1. Write down the problem.
2. Think real hard.
3. Write down the solution.

Binary search

Use if the input is sorted.

If the input consists of numbers, for example, half the numbers are eliminated in each step with binary search. When searching for a number between 1 and 100, a maximum of 7 steps are needed to get the result.

100 -> 50 -> 25 -> 13 -> 7 -> 4 -> 2 -> 1

In general, for any list of n, binary search will take log₂n steps to run in the worst case.

def binary_search(_list, item):
    high = len(_list) - 1
    low = 0

    while high >= low:
        mid = (high + low) // 2
        guess = _list[mid]
        if guess == item:
            return mid
        elif guess > item:
            high = mid - 1
        else:
            low = mid + 1

    return None


if __name__ == "__main__":
    position = binary_search([2, 3, 5, 7, 8, 9, 12, 24, 32], 12)
    print(position)

Arrays & linked lists

Arrays

Random access. Fast reads, slow writes.

With arrays, data is stored contiguously (right next to each other) in memory. If more data needs to be added and there’s not enough space for it to be stored contiguously in the current location, everything needs to be moved to a different location that will allow this to be done. Inserting data into the middle of an array also requires moving everything else to the right to make space for the new item. Adding data can therefore be quite slow. While it’s possible to reserve slots as a workaround for having to change locations, the reserved slots end up being wasted in case they’re not used. Deleting is also slow as everything needs to be moved up to fill the empty space (assuming the deletion was done in the middle).

Linked lists

Sequential access. Fast writes, slow reads.

Data can be saved anywhere in memory. Each item stores the address of the next item in the list. This solves the problem of needing to store everything together, but is slow when reading data. To read the last item in the list, all previous items will have to be accessed. Adding and deleting is fast as it only requires a change in the next address pointed to.

Recursion

Loops may achieve a performance gain for your program. Recursion may achieve a performance gain for your programmer. Choose which is more important in your situation! - https://stackoverflow.com/a/72694/9312256

Every recursive function has two parts: the base case, and the recursive case.

Tip 💡

When you’re writing a recursive function involving an array, the base case is often an empty array or an array with one element.

Exercise

Find the total of a list of numbers using recursion.

def _sum(_list):
    if not _list:
        return 0

    if len(_list) == 1:
        return _list[0]
    else:
        return _list.pop() + _sum(_list)

print(_sum([1,2,3,4,5])) # => 15
print(_sum([2,4,6])) # => 12

The stack

Items are added at the top (push) and removed from the top (pop). FILO.

The call stack with recursion

Example using factorial calculation:

Code:

def fact(x):
  if x == 1:
    return 1
  else:
    return x * fact(x-1)

fact(3)

Call stack:

Each call to fact has its own copy of x. You can’t access a different function’s copy of x.

Divide & conquer

D&C algorithms are recursive algorithms.

To solve a problem using D&C, there are two steps:

1. Figure out the base case. This should be the simplest possible case.
2. Divide or decrease your problem until it becomes the base case.

Exercise

Suppose you’re a farmer with a plot of land - 1680 * 640 meters. You want to divide this farm evenly into square plots. You want the plots to be as big as possible. How do you figure out the largest square size you can use for a plot of land?

def divide_land(a, b):
    bigger_side = max(a,b)
    smaller_side = min(a,b)

    if smaller_side * 2 == bigger_side:
        return (smaller_side, smaller_side)
    else:
        rem = bigger_side % smaller_side
        return divide_land(rem, smaller_side)

print(divide_land(1680, 640)) # => (80, 80)
print(divide_land(400, 640)) # => (80, 80)

Quicksort

Uses divide & conquer.

Sorting an array of numbers with quick sort

Base case: empty array or array with one element.

For bigger arrays, choose an element from the array (pivot) and find elements smaller than the pivot and larger than it. All smaller elements go to the left of the pivot. Bigger elements go to the right of the pivot. Recurse until array is sorted.

def quicksort(array):
    if len(array) < 2:
        return array
    pivot = array[0]
    less = [i for i in array[1:] if i < pivot]
    equal = [i for i in array[1:] if i == pivot]
    greater = [i for i in array[1:] if i > pivot]

    return quicksort(less) + equal + [pivot] + quicksort(greater)


print(quicksort([10, 2, 3, 7, 12, 2, 5, 1, 3])) # => [1, 2, 2, 3, 3, 5, 7, 10, 12]

The performance of quicksort heavily depends on the pivot choosen. If the first element is the pivot and quicksort is called with an array that is already sorted it doesn’t check to see whether the input array is already sorted. So it will still try to sort it.

Middle pivots work better. In the worst case (first item as pivot), the stack size is O(n). In the best case (middle pivot), the stack size is O(log n).

The overall complexity of quicksort is O(n log n)

Graphs

A graph models a set of connections / how different things are connected to one another. It’s made up of nodes and edges. A node can be directly connected to many other nodes. Those nodes are called its neighbors.

Breadth-first search

Used to find the shortest path to a target. It can help answer two types of questions:

• Is there a path from node A to node B?
• What is the shortest path from node A to node B?

Example

Suppose you’re the proud owner of a mango farm. You’re looking for a mango seller who can sell your mangoes. Are you connected to a mango seller on Facebook? Well, you can search through your friends.

First, make a list of friends to search. Now, go to each person in the list and check whether that person sells mangoes. Suppose none of your friends are mango sellers. Now you have to search through your friends’ friends. Each time you search for someone from the list, add all of their friends to the list. This way, you not only search your friends, but you search their friends too.

First should be searched first before friends of friends. To keep this order, you need a queue.

from collections import deque

graph = dict()
graph["you"] = ["alice", "bob", "claire"]
graph["bob"] = ["anuj", "peggy"]
graph["alice"] = ["peggy"]
graph["claire"] = ["tom mango seller", "jonny"]
graph["anuj"] = []
graph["peggy"] = []
graph["tom mango seller"] = []
graph["jonny"] = []


def search(name):
    search_queue = deque()
    search_queue += graph[name]
    searched = []

    while search_queue:
        person = search_queue.popleft()
        if not person in searched:
            if "mango seller" in person:
                return person
            else:
                search_queue += graph[person]
                searched.append(person)
    return False

print(search("you")) # => tom mango seller

Running time: O(V+E) (V for number of vertices, E for number of edges).

Tip 💡

If you have a problem like “find the shortest X,” try modeling your problem as a graph and use breadth-first search to solve.

Dijkstra’s algorithm

BFS gets the shortest path to a target, in terms of number of nodes that have to be accessed before reaching the target. Dijkstra’s algorithm gets the fastest path to a target, even though the path may not necessarily be the shortest.

To calculate the shortest path in an unweighted graph, use breadth-first search. To calculate the shortest path in a weighted graph, use Dijkstra’s algorithm.

Steps:

1. Find the “cheapest” node. This is the node that can be accessed in the least amount of time.
2. Update the costs of the neighbors of this node.
3. Repeat until this has been done for every node in the graph.
4. Calculate the final path.

# TODO: implementation

Note 🐛

You can’t use Dijkstra’s algorithm if you have negative-weight edges as they break the algorithm.

For a graph with negative-weight edges, use Bellman-Ford algorithm.

Greedy algorithms

A greedy algorithm is simple: at each step, pick the optimal move. In technical terms: at each step, pick the locally optimal solution, and in the end you’re left with the globally optimal solution. It doesn’t always get the most optimal solution, but it comes close enough.

Dynamic programming

Dynamic programming starts by solving subproblems and builds up to solving the big problem.

Useful when trying to optimize something given a constraint.

import (
    "fmt"
    "regexp"
    "strings"
)

const (
    str    = "something"
    substr = "some"
)

// 1. Contains
res := strings.Contains(str, substr)
fmt.Println(res) // true

// 2. Index: check the index of the first instance of substr in str, or -1 if substr is not present
i := strings.Index(str, substr)
fmt.Println(i) // 0

// 3. Split by substr and check len of the slice, or length is 1 if substr is not present
ss := strings.Split(str, substr)
fmt.Println(len(ss)) // 2

// 4. Check number of non-overlapping instances of substr in str
c := strings.Count(str, substr)
fmt.Println(c) // 1

// 5. RegExp
matched, _ := regexp.MatchString(substr, str)
fmt.Println(matched) // true

// 6. Compiled RegExp
re = regexp.MustCompile(substr)
res = re.MatchString(str)
fmt.Println(res) // true

Option two easily solves this problem.

>>> d = {}
>>> d
{}
>>> d["x"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'x'
>>> d.get("x")
>>> d
{}
>>> d.get("x","y")
'y'
>>> d
{}
>>>

But what if you want to not only return a default if the key is missing, but also assign that default to the specified key in the dict? There are a couple of ways to do that.

1. setdefault

setdefault(key[, default])

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

docs

>>> d
{}
>>> d.setdefault("x","y")
'y'
>>> d
{'x': 'y'}
>>>

2. defaultdict

class collections.defaultdict(default_factory=None, /[, ...])

The first argument provides the initial value for the default factory attribute; it defaults to None. All remaining arguments are treated the same as if they were passed to the dict constructor, including keyword arguments.

docs

>>> from collections import defaultdict
>>> i = defaultdict(int)
>>> i
defaultdict(<class 'int'>, {})
>>> i["x"]
0
>>> i
defaultdict(<class 'int'>, {'x': 0})
>>> i["y"]
0
>>> i
defaultdict(<class 'int'>, {'x': 0, 'y': 0})
>>>

Note that no KeyError is raised despite the key not existing at first. Instead, the key is created with the default value of the type passed into defaultdict, in this case 0 for int. If we used list instead, the default value would be [], and so on.

Further reading:

• defaultdict examples
• Use cases for the ‘setdefault’ dict method (vs defaultdict)

3. Implement __missing__

This is actually what defaultdict does behind the scenes. Use this when defaultdict doesn’t fit your usecase.

>>> class M(dict):
...     def __missing__(self, key):
...             value = "my default value"
...             self[key] = value
...             return value
...
>>>
>>> m = M()
>>> m
{}
>>> m["x"]
'my default value'
>>> m
{'x': 'my default value'}
>>> m["y"]
'my default value'
>>> m
{'x': 'my default value', 'y': 'my default value'}
>>>

Steps

1. Initialize two variables: max_sum = current_sum = 0
2. Loop through numbers in list
   • Set current_sum = max(0, current_sum + list[i])
   • Set max_sum = max(max_sum, current_sum)
3. Return max_sum

Caveats ¹, ²

• Kadane’s Algorithm requires at least one positive number, so an input of all negative numbers would be invalid.
• If the array contains all non-negative numbers, then the problem is trivial; a maximum subarray is the entire array.
• If the array contains all non-positive numbers, then a solution is any subarray of size 1 containing the maximal value of the array (or the empty subarray (which has sum 0), if it is permitted).
• Several different sub-arrays may have the same maximum sum.

Solving Best Time to Buy and Sell Stock with Kadane’s Algorithm

Slightly tweak the algorithm to track max profit and min price.

def maxProfit(self, prices: List[int]) -> int:
    max_profit, min_price = 0, float("inf")
    for price in prices:
        min_price = min(min_price, price)
        max_profit = max(max_profit, price - min_price)
    return max_profit

• run a MySQL proxy to Cloudql or similar, then
• run a project server (fastapi, django etc) that depends on the database connection

From this answer:

set -m  # enable job control

prog1 & prog2 && fg

This will:

• Start prog1.
• Send it to background, but keep printing its output.
• Start prog2, and keep it in foreground, so you can close it with ctrl-c.
• When you close prog2, you’ll return to prog1’s foreground, so you can also close it with ctrl-c.

zip(*iterables, strict=False)

Iterate over several iterables in parallel, producing tuples with an item from each one.

Examples

Without zip

countries = ["Kenya", "Tanzania", "Uganda"]
capitals = ["Nairobi", "Dodoma", "Kampala"]

print("=== using range ===")
for i in range(len(countries)):
    country, capital = countries[i], capitals[i]
    print(f"Country: {country}, Capital: {capital}")

print("=== using enumerate ===")
for i, country in enumerate(countries):
    country, capital = countries[i], capitals[i]
    print(f"Country: {country}, Capital: {capital}")

Output:

=== using range ===
Country: Kenya, Capital: Nairobi
Country: Tanzania, Capital: Dodoma
Country: Uganda, Capital: Kampala

=== using enumerate ===
Country: Kenya, Capital: Nairobi
Country: Tanzania, Capital: Dodoma
Country: Uganda, Capital: Kampala

With zip

print("=== using zip ===")
for country, capital in zip(countries, capitals):
    print(f"Country: {country}, Capital: {capital}")

Output:

=== using zip ===
Country: Kenya, Capital: Nairobi
Country: Tanzania, Capital: Dodoma
Country: Uganda, Capital: Kampala

Same result, but more elegant.

import sys

print(sys.version)

Output:

3.9.7 (default, Aug 31 2021, 13:28:12)
[GCC 11.1.0]

I don’t know if it’s very useful but definitely interesting.

MariaDB[] > SELECT SIN(PI()/4), (4+1)*5;
+------------------+---------+
| SIN(PI()/4)      | (4+1)*5 |
+------------------+---------+
| 0.70710678118655 |      25 |
+------------------+---------+
1 row in set (0.02 sec)

postgres=# SELECT SIN(PI()/4), (4+1)*5;
        sin         | ?column?
--------------------+----------
 0.7071067811865475 |       25
(1 row)

Install

$ sudo pacman -S postgresql

Setup the data directory

$ sudo -iu postgres
[postgres]$ initdb -D /var/lib/postgres/data

Start postgresql.service

$ systemctl start postgresql.service
$ systemctl enable postgresql.service

Add user

[postgres]$ createuser --interactive

Create db

$ createdb myDatabaseName

MariaDB is a reliable, high performance and full-featured database server which aims to be an ‘always Free, backward compatible, drop-in’ replacement of MySQL. Since 2013 MariaDB is Arch Linux’s default implementation of MySQL.

Install

$ sudo pacman -S mariadb

Setup the data directory

$ sudo mariadb-install-db --user=mysql --basedir=/usr --datadir=/var/lib/mysql

Start mariadb.service

$ systemctl start mariadb.service
$ systemctl enable mariadb.service

Add user

In the below example, the user monty with some_pass as password is being created, then granted full permissions to the database mydb:

$ sudo mysql -u root -p

MariaDB> CREATE USER 'monty'@'localhost' IDENTIFIED BY 'some_pass';
MariaDB> GRANT ALL PRIVILEGES ON mydb.* TO 'monty'@'localhost';
MariaDB> FLUSH PRIVILEGES;
MariaDB> quit

Logging into MariaDB

mysql -u //user_name// -p -h //ip_address// //db_name//

Eg:

mysql -u username -ppassword -h localhost database_name

package main

import (
    "log"
    "runtime"
)

func main() {
    log.Printf("Running Go: %s", runtime.Version())
}

Output: Running Go: go1.16.6

This SO answer sums it up nicely:

Readability is more important for tests. If a test fails, you want the problem to be obvious. The developer shouldn’t have to wade through a lot of heavily factored test code to determine exactly what failed. You don’t want your test code to become so complex that you need to write unit-test-tests.

However, eliminating duplication is usually a good thing, as long as it doesn’t obscure anything, and eliminating the duplication in your tests may lead to a better API. Just make sure you don’t go past the point of diminishing returns.

pytest gives some ways to reduce duplication with parametrize.

Say you had a couple of endpoints that return data to be used in a report. The report data is supposed to be displayed in an excel sheet with different sheets.

Sometimes, only data for one sheet is required. Other times, data for all the sheets is fetched. So the endpoints end up being broken down like this:

/report/sheet-a
/report/sheet-b
/report/sheet-c
/report/sheet-d

Let’s take a simple test case: checking that the endpoints return 200 when called.

import pytest

def test_report_sheet_a_returns_200():
    response = client.get('/report/sheet-a')
    assert response.status == 200

def test_report_sheet_b_returns_200():
    response = client.get('/report/sheet-b')
    assert response.status == 200

def test_report_sheet_c_returns_200():
    response = client.get('/report/sheet-c')
    assert response.status == 200

def test_report_sheet_d_returns_200():
    response = client.get('/report/sheet-d')
    assert response.status == 200

This might not look too bad, but if we wanted to test for another operation on the endpoints - checking if the endpoints require authentication for example - you get the feeling that this can be done a little bit better.

import pytest

# test GET requests

def test_report_sheet_a_returns_200():
    response = client.get('/report/sheet-a')
    assert response.status == 200

def test_report_sheet_b_returns_200():
    response = client.get('/report/sheet-b')
    assert response.status == 200

def test_report_sheet_c_returns_200():
    response = client.get('/report/sheet-c')
    assert response.status == 200

def test_report_sheet_d_returns_200():
    response = client.get('/report/sheet-d')
    assert response.status == 200

# test auth

def test_report_sheet_a_requires_auth():
    response = unauthorized_client.get('/report/sheet-a')
    assert response.status == 401

def test_report_sheet_b_requires_auth():
    response = unauthorized_client.get('/report/sheet-b')
    assert response.status == 401

def test_report_sheet_c_requires_auth():
    response = unauthorized_client.get('/report/sheet-c')
    assert response.status == 401

def test_report_sheet_d_requires_auth():
    response = unauthorized_client.get('/report/sheet-d')
    assert response.status == 401

Notice that the only thing changing in the tests is the endpoints. Everything else remains the same.

Given that all the sheets belong to one report, we can refactor the tests to reduce duplication without sacrificing readability.

Here’s how the tests can be rewritten using Pytest parametrize:

import pytest

report_sheet_endpoints = (
  '/sheet-a',
  '/sheet-b',
  '/sheet-c',
  '/sheet-d',
)


@pytest.mark.parametrize('endpoint', report_sheet_endpoints)
def test_report_sheets_return_200(endpoint):
    response = client.get(f'/report{endpoint}')
    assert response.status == 200


@pytest.mark.parametrize('endpoint', report_sheet_endpoints)
def test_report_sheets_require_auth(endpoint):
    response = unauthorized_client.get(f'/report{endpoint}')
    assert response.status == 401

Now we have two tests instead of eight. But when you run the tests, 8 tests will run, not 2. Pytest takes each value in report_sheet_endpoints and feeds it into the test. This reduces duplication while maintaining readability.

Running the tests gives this output:

test_reports.py::test_report_sheets_return_200[sheet-a] PASSED
test_reports.py::test_report_sheets_return_200[sheet-b] PASSED
test_reports.py::test_report_sheets_return_200[sheet-c] PASSED
test_reports.py::test_report_sheets_return_200[sheet-d] PASSED

test_reports.py::test_report_sheets_require_auth[sheet-a] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-b] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-c] PASSED
test_reports.py::test_report_sheets_require_auth[sheet-d] PASSED

$ psql
psql (13.0)
Type "help" for help.

wangonya=# \?
General
  \copyright             show PostgreSQL usage and distribution terms
  \crosstabview [COLUMNS] execute query and display results in crosstab
  \errverbose            show most recent error message at maximum verbosity

  .....

But what are the queries executed when these commands run?

I was recently tasked to write a bash script that loops through every schema on a database and does a pg_dump on it for backup then uploads the backups to an external server.

Running \dn gives a list of schemas but it also gives the owners, which I didn’t need in my case.

wangonya=# \dn
  List of schemas
  Name  |  Owner
--------+----------
 public | wangonya
 x      | wangonya
 y      | wangonya
(3 rows)

I only needed the schema names so I had to look for a way to edit the query run by \dn to only return names.

From the psql docs:

-E > --echo-hidden

Echo the actual queries generated by \d and other backslash commands. You can use this to study psql’s internal operations. This is equivalent to setting the variable ECHO_HIDDEN to on.

Running psql with the -E flag then running the /dn command gives this result:

$ psql -E

wangonya-# \dn
********* QUERY **********
SELECT n.nspname AS "Name",
  pg_catalog.pg_get_userbyid(n.nspowner) AS "Owner"
FROM pg_catalog.pg_namespace n
WHERE n.nspname !~ '^pg_' AND n.nspname <> 'information_schema'
ORDER BY 1;
**************************

  List of schemas
  Name  |  Owner
--------+----------
 public | wangonya
 x      | wangonya
 y      | wangonya
(3 rows)

With the underlying query executed on running the slash command, I was able to edit it to get the desired result.

Sometimes when using celery, you may want to get notified when a task running in the background executes successfully or when it fails. You may also want to run a function each time before the celery task runs or after it completes. These and many others along the same line are all situations where signals would come in handy.

I’ll create a simple file tasks.py and set up celery to demonstrate how to use celery signals.

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

Make sure your redis server is running and start your celery worker:

(env) $ celery -A tasks worker --loglevel=INFO

Then run your tasks.py file and execute the add task:

(env) $ python -i tasks.py

>>> add.delay(4, 4)
<AsyncResult: ce1ee079-6434-4f54-ace2-360ff316546b>
>>>

By default, what is returned is an AsyncResult instance but that’s not what we’re interested in. On the terminal with your Celery worker running, you should see something similar to this:

[2020-11-03 07:01:02,024: INFO/ForkPoolWorker-2] Task tasks.add[ce1ee079-6434-4f54-ace2-360ff316546b] succeeded in 0.0005510429999979749s: 8

The task executes successfully, and 8 is the result as expected.

Signals

There are a lot of signals that celery offers but I’ll focus on 4 simple ones to demonstrate how signals work in general.

1. task_prerun
2. task_postrun
3. task_success
4. task_failure

task_prerun

This signal is dispatched before a task is executed.

from celery import Celery
from celery.signals import task_prerun

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

@task_prerun.connect(sender=add)
def task_prerun_notifier(sender=None, **kwargs):
    print("From task_prerun_notifier ==> Running just before add() executes")

Sender is the task object being executed (the add function in this case).

Running add.delay(4, 4) like before now gives the following output on the celery terminal:

[2020-11-03 07:23:19,183: WARNING/ForkPoolWorker-2] From task_prerun_notifier ==> Running just before add() executes
[2020-11-03 07:23:19,184: INFO/ForkPoolWorker-2] Task tasks.add[1ef11c46-f461-4eb8-84ca-5c5cdab62a74] succeeded in 0.0016491969999998801s: 8

Just before the task runs, the signal dispatches and prints as expected.

task_postrun

Dispatched after a task has been executed.

from celery.signals import task_prerun, task_postrun

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

@task_prerun.connect(sender=add)
def task_prerun_notifier(sender=None, **kwargs):
    print("From task_prerun_notifier ==> Running just before add() executes")

@task_postrun.connect(sender=add)
def task_postrun_notifier(sender=None, **kwargs):
    print("From task_postrun_notifier ==> Ok, done!")

Running this should give the following result:

[2020-11-03 17:03:51,655: WARNING/ForkPoolWorker-2] From task_prerun_notifier ==> Running just before add() executes
[2020-11-03 17:03:51,656: INFO/ForkPoolWorker-2] Task tasks.add[7da6ee71-1941-4a87-b993-8136d94ac067] succeeded in 0.0017917519999999243s: 8
[2020-11-03 17:03:51,657: WARNING/ForkPoolWorker-2] From task_postrun_notifier ==> Ok, done!

task_success

Dispatched when a task succeeds.

from celery import Celery
from celery.signals import task_prerun, task_postrun, task_success

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    return x + y

@task_prerun.connect(sender=add)
def task_prerun_notifier(sender=None, **kwargs):
    print("From task_prerun_notifier ==> Running just before add() executes")

@task_postrun.connect(sender=add)
def task_postrun_notifier(sender=None, **kwargs):
    print("From task_postrun_notifier ==> Ok, done!")

@task_success.connect(sender=add)
def task_success_notifier(sender=None, **kwargs):
    print("From task_success_notifier ==> Task run successfully!")

Result:

[2020-11-03 17:40:47,276: INFO/MainProcess] Received task: tasks.add[6603eb49-75ab-4653-b32f-ebe760a52de0]
[2020-11-03 17:40:47,279: WARNING/ForkPoolWorker-2] From task_prerun_notifier ==> Running just before add() executes
[2020-11-03 17:40:47,281: WARNING/ForkPoolWorker-2] From task_success_notifier ==> Task run successfully!
[2020-11-03 17:40:47,281: INFO/ForkPoolWorker-2] Task tasks.add[6603eb49-75ab-4653-b32f-ebe760a52de0] succeeded in 0.00201471799999986s: 8
[2020-11-03 17:40:47,282: WARNING/ForkPoolWorker-2] From task_postrun_notifier ==> Ok, done!

task_failure

Dispatched when a task fails.

from celery import Celery
from celery.signals import task_prerun, task_postrun, task_failure

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    raise Exception

@task_prerun.connect(sender=add)
def task_prerun_notifier(sender=None, **kwargs):
    print("From task_prerun_notifier ==> Running just before add() executes")

@task_postrun.connect(sender=add)
def task_postrun_notifier(sender=None, **kwargs):
    print("From task_postrun_notifier ==> Ok, done!")

@task_failure.connect(sender=add)
def task_failure_notifier(sender=None, **kwargs):
    print("From task_failure_notifier ==> Task failed successfully! 😅")

Result:

[2020-11-03 17:44:36,082: INFO/MainProcess] Received task: tasks.add[da4a03e8-5530-4c9e-afeb-75f8e0b1be5d]
[2020-11-03 17:44:36,085: WARNING/ForkPoolWorker-2] From task_prerun_notifier ==> Running just before add() executes
[2020-11-03 17:44:36,096: WARNING/ForkPoolWorker-2] From task_failure_notifier ==> Task failed successfully! 😅
[2020-11-03 17:44:36,096: ERROR/ForkPoolWorker-2] Task tasks.add[da4a03e8-5530-4c9e-afeb-75f8e0b1be5d] raised unexpected: Exception()
Traceback (most recent call last):
  ...
   in add
    raise Exception
Exception
[2020-11-03 17:44:36,097: WARNING/ForkPoolWorker-2] From task_postrun_notifier ==> Ok, done!

>>> x = 5
>>> y = x

Both x and y will have the value 5. But, when x was assigned to y, y was not created as a completely new/separate object. Instead, an alias for x was created. That is, y points to the memory location of x. It does not have its own memory location - yet.

>>> id(x)
140428600776960
>>> id(y)
140428600776960
>>> x is y
True

You may never have any problems with this when working with immutable types because the alias is broken as soon as either of the two variables change.

>>> x += 2
>>> x
7
>>> y
5
>>> id(x)
140539682924864
>>> id(y)
140428600776960
>>> x is y
False

But when working with mutable types, the alias is not broken when the original is updated. This means changes in x would reflect in y.

>>> x = [1,2,3]
>>> y = x
>>> x
[1, 2, 3]
>>> y
[1, 2, 3]
>>> x is y
True
>>>
>>> x.append(4)
>>> x.append(5)
>>> x
[1, 2, 3, 4, 5]
>>> y
[1, 2, 3, 4, 5]
>>> x is y
True

y was updated externally. This might be a cause of bugs if for example a value is updated by an external library and other variables are affected.

This can be prevented by creating shallow/deep copies of objects instead of using assignment.

Shallow copy

A shallow copy creates a new object, then populates it with references of the objects in the original object. Continuing with the previous example, a shallow copy can be created using either the list() or copy() command.

>>> z = list(x)
>>> z
[1, 2, 3, 4, 5]

Now if some more values are appended to x, y will still be affected, but z will not.

>>> x.append(6)
>>> x.append(7)
>>> x
[1, 2, 3, 4, 5, 6, 7]
>>> y
[1, 2, 3, 4, 5, 6, 7]
>>> z
[1, 2, 3, 4, 5]

However, a shallow copy doesn’t fully solve the problem because even though a new list was created, the objects in the list are still references to the objects in x.

As it is currently, updating x[0] would not affect z[0] because - immutable objects - the alias would be broken. But, if we were dealing with a list of lists, an update in x[0] would affect z[0].

>>> x = [[1,2], [3,4]]
>>> y = x
>>> z = list(x)
>>> x
[[1, 2], [3, 4]]
>>> y
[[1, 2], [3, 4]]
>>> z
[[1, 2], [3, 4]]
>>> x.append([5,6])
>>> x
[[1, 2], [3, 4], [5, 6]]
>>> y
[[1, 2], [3, 4], [5, 6]]
>>> z
[[1, 2], [3, 4]]
>>>
>>> x[0][1] = 'edited'
>>> x
[[1, 'edited'], [3, 4], [5, 6]]
>>> y
[[1, 'edited'], [3, 4], [5, 6]]
>>> z
[[1, 'edited'], [3, 4]]

Deep copy

A deep copy creates a new object, and completely new instances of the objects in it. That is, a deep copied object is completely independent of the original. Updating objects in the original would not affect the deep copied object since there’s no longer any connection.

>>> import copy
>>> x = [[1,2], [3,4]]
>>> z = copy.deepcopy(x)
>>> x
[[1, 2], [3, 4]]
>>> z
[[1, 2], [3, 4]]
>>> x is z
False
>>> x[0][1] = 'edited'
>>> x
[[1, 'edited'], [3, 4]]
>>> z
[[1, 2], [3, 4]]

fn main() {
    let var = 10;
    println!("var = {}", var);  // var = 10
    var = 5;  // try and reassign var
    println!("var is now {}", var);
}

… will not compile.

The rust compiler gives a clear output of what went wrong:

error[E0384]: cannot assign twice to immutable variable `var`
  |
2 |     let var = 10;
  |         ---
  |         |
  |         first assignment to `var`
  |         help: make this binding mutable: `mut var`
3 |     println!("var = {}", var);  // var = 10
4 |     var = 5;  // try and reassign var
  |     ^^^^^^^ cannot assign twice to immutable variable

Once a value is assigned to var, var is bound to that value until it goes out of scope.

Opting out of immutability

Variables can be made mutable using the mut keyword:

fn main() {
    let mut var = 10;
    println!("var = {}", var);  // var = 10
    var = 5;  // reassign var
    println!("var is now {}", var);  // var is now 5
}

This makes it clear when reading the code that var expects to be changed somewhere else.

Shadowing

Instead of making a variable mutable and reassigning it directly, rust provides another technique known as shadowing. Here’s an example:

fn main() {
    let var = 10;  // not mutable
    println!("var = {}", var);  // var = 10
    let var = 5;  // shadow first var (10)
    println!("var is now {}", var);  // var is now 5
    let var = var + 2;  // shadow second var (5) and add 2 to it
    println!("var + 2 = {}", var);  // var + 2 = 7
}

var remains immutable all through so trying to attach any value to it without using let to tell Rust you’re shadowing will not work.

Shadowing also allows changing the variable data type - something that making a variable mutable doesn’t allow. An example of when to use this would be when writing a program that accepts a string and returns the number of characters in that string.

This works:

let hello = "hello";
let hello = hello.len();  // 5

This will not compile:

let mut hello = "hello";
hello = hello.len();

Output:

error[E0308]: mismatched types
  |
3 |     hello = hello.len();
  |             ^^^^^^^^^^^ expected `&str`, found `usize`

Variables vs constants

• Constants are declared with const keyword, variables with let
• Constants cannot be made mutable
• Constants can only be assigned to constant expressions, not to values that are computed during runtime - like results of function calls

for n in range(50):
    ...

#

for n in range(10, 30):
    ...

or a list of strings:

for fruit in ["apple", "mango", "banana"]:
    ...

Now, say you want to iterate over the list of fruits and also show the index of the current item in the list. Using range(), this might be done like this:

fruits = ["apple", "mango", "banana"]

for i in range(len(fruits)):
    fruit = fruits[i]
    print(f"{i}: {fruit}")

# Output:
# 0: apple
# 1: mango
# 2: banana

It gets the job done, but not very pythonic. You have to get the length of the list to keep track of the index, then index into the array to get the current fruit - which makes the code more verbose and harder to read.

A better way: enumerate()

fruits = ["apple", "mango", "banana"]

for i, fruit in enumerate(fruits):
    print(f"{i}: {fruit}")

# Output:
# 0: apple
# 1: mango
# 2: banana

Numbering can also be set to begin at any desired number.

fruits = ["apple", "mango", "banana"]

for i, fruit in enumerate(fruits, 7):
    print(f"{i}: {fruit}")

# Output
# 7: apple
# 8: mango
# 9: banana

Python has a built-in package called difflib with the function get_close_matches() that can help us.

get_close_matches(word, possibilities, n, cutoff) accepts four parameters:

• word - the word to find close matches for in our list
• possibilities - the list in which to search for close matches of word
• n (optional) - the maximum number of close matches to return. Must be > 0. Default is 3.
• cutoff (optional) - a float in the range [0, 1] that a possibility must score in order to be considered similar to word. 0 is very lenient, 1 is very strict. Default is 0.6.

An example from the docs:

Python 3.7.3

>>> from difflib import get_close_matches
>>> get_close_matches('appel', ['ape', 'apple', 'peach', 'puppy'])
['apple', 'ape']

The above example can easily be modified to use a custom list _list for possibilities and user input _input for word.

Just paste in your HTML in the HTML editor on the left, and it’ll be converted into valid JSX on the right.

# do I do
if a is b:
    ...

# or
if a == b:
    ...

It can be a bit confusing if you’re new to Python, and it’s easy to assume the two can be used interchangeably. So, what’s the difference?

is

The is operator checks if both elements point to the same object. Let’s fire up a python console to help illustrate this:

$ python3
Python 3.7.4
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> a = []
>>> b = []
>>> c = a

>>> a
[]
>>> b
[]
>>> c
[]
>>>

So, we’ve declared three variables and assigned them values. a and b are both empty lists, and c = a. We can see that all three variables contain an empty list. Using is to compare them:

>>> a is b
False
>>> b is c
False
>>> a is c
True

Despite the fact that a and b seem identical (in that they’re both empty lists), the variables a and b do not point to the same object, therefore a is b evaluates to False. The same goes for b is c.

Conversely, because we assigned the variable a to c, they both point to the same object, thus a is c is True.

==

== on the other hand checks if both elements contain equal values. Whether or not they point to the same object doesn’t matter here.

>>> a == b
True
>>> b == c
True
>>> a == c
True

All checks using == evaluate to True, because the values of a, b and c are all equal. If d = [1, 2, 3] is introduced, a == d, b == d and c == d would all be False, because the values are not equal.

So if you want to check that elements point to the same object, use is. If you’re only interested in the equality of the values, use ==.

You may already know about SMART goals, which is an awesome format to set your goals in. It does have a subtle pitfall though: cognitive closure.

Cognitive closure

Cognitive closure (psychology), the human desire to eliminate ambiguity and arrive at definite conclusions (sometimes irrationally) - Wikipedia

You know that nice feeling of crossing things off your to-do list? Closure. Which is great, for the most part. It only gets bad when we end up expending more effort to get immediate results, when we are so focused on small daily "wins" that we lose focus on the larger picture.

“You get into this mindset where crossing things off your to-do list becomes more important than asking yourself if you’re doing the right things. - The book

"To-do lists are great if you’re using them correctly. But when people say things like ‘I sometimes write down easy items I can cross off right away, because it makes me feel good’, that’s exactly the wrong way to create a to-do list. That signals you’re using it for mood repair, rather than to become productive." - Timothy Pychyl, quoted in the book

I think it’s easier to fall into this trap than most of us would like to admit.

Combining SMART goals with Stretch goals

Stretch goals are goals that may be considered audacious, but achievable. These are the kind of goals you may think about when asked to "dream big". The point is to have one major goal that is not immediately achievable, then use the SMART strategy to form sub-goals that move you closer to achieving the main goal. Here’s an example from the book:

• Stretch goal: run a marathon
• Specific sub-goal: run 7 miles without stopping
• Measurable: run twice around the park, no walking
• Achievable?: yes, if I run 3 times a week
• Realistic?: yes, if I wake up early on Mondays, Wednesdays, and Fridays
• Timeline: run 3 miles this week, 4 miles next week, 5 miles…

Setting goals in this format ensures that we do get those small daily wins, and always stay focused on the big picture.

How to use CSS counters

Getting counters working requires three steps:

• Initialize the counter with counter-reset
• Increment the counter with counter-increment
• Show the current counter value with counter()

It’s pretty simple, really. The Pen below demonstrates how to count and number three <div> elements.

Where would this be useful?

I discovered CSS counters while looking for a way to number images on my blog posts. That’s one place it would really come in handy. Here’s a quick example:

a separate instance of the Python Package Index that allows you to try distribution tools and processes without affecting the real index.

Uploading to TestPyPi

For the final part of this series, I’ll upload the app we created on TestPyPi. Here are the steps to follow:

1. Create an account on TestPyPi

You’ll need an account to continue so head over to https://test.pypi.org/ and create one.

2. Update setup.py

We kept the setup.py file very simple during development. Now we’ll need to add some details to make it ready for publishing.

from setuptools import setup, find_packages

with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
        name="contacts",
        version="0.1.0",
        packages=find_packages(),
        description="Save contacts from your terminal",
        long_description=long_description,
        long_description_content_type="text/markdown",
        url="https://github.com/wangonya/contacts-cli",
        author="Kelvin Wangonya",
        author_email="kwangonya@gmail.com",
        license="MIT",
        classifiers=[
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.7",
        ],
        install_requires=[
            "Click",
            "requests",
            ],
        entry_points="""
        [console_scripts]
        contacts=app:cli
        """,
        )

Most of the options are self-explanatory, but you can check what each of them means in the setup docs.

3. Build your package

In the root directory, run:

(env) $ python setup.py sdist bdist_wheel

This will generate a dist folder with the distribution packages:

dist/
├── contacts-0.1.0-py3-none-any.whl
└── contacts-0.1.0.tar.gz

4. Upload your package with twine

Install twine:

$ pip install twine

Upload your package:

$ twine upload --repository-url https://test.pypi.org/legacy/ dist/*
Enter your username: wangonya
Enter your password:
Uploading distributions to https://test.pypi.org/legacy/
Uploading contacts-0.1.0-py3-none-any.whl
100%|███████████████████████████████████████████████████████████|
Uploading contacts-0.1.0.tar.gz
100%|███████████████████████████████████████████████████████████|

The package should now be uploaded on your account.

To install the package locally from TestPyPi, run:

$ pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple your-package

--index-url tells pip to get the package from TestPyPi. If the package needs other dependencies to run, --extra-index-url has to be included so the dependencies can be fetched from the PyPi.

• Getting started with pytest
• Asserting Exceptions with Pytest
• Capturing print statements while debugging
• Skipping tests

For testing CLI apps, Click provides a convenient module: click.testing which has some useful functions (notably CliRunner()) to help us invoke commands and check their behavior.

We’ll go ahead and test each part of our app - creating, reading, updating and deleting.

Installing pytest and writing the first test

pytest can be installed via pip:

(env) $ pip install pytest

After installing pytest, create a tests folder in the root directory and add the first test file:

(env) $ mkdir tests && cd tests

(env) $ touch test_app.py

In the test_app file, add the following code for a start:

def test_add():
    pass

To run the test, run pytest on the terminal:

(env) $ pytest
================== test session starts ====================
platform linux -- Python 3.7.3, pytest-5.1.0, py-1.8.0, pluggy-0.12.0
rootdir: /home/wangonya/code/contacts-cli
collected 1 item

tests/test_app.py .                              [100%]

================== 1 passed in 0.04s =======================

Testing the add command

Let’s edit the test_app file to add a test to see if the add command adds a new contact:

from click.testing import CliRunner

from app import add

runner = CliRunner()

def test_add():
    response = runner.invoke(add, ["test-user", "-m", "0"])
    assert response.exit_code == 0
    assert "Contact test-user added!" in response.output
    assert "{'mobile': '0'}" in response.output

First, we invoke the command as we would on the terminal, passing in the required arguments and options: response = runner.invoke(add, ["test-user", "-m", "0"]).

We then check that the command executes successfully: assert response.exit_code == 0.

If the command executes successfully, we expect a success message should be returned in the response with the values we added:

assert "Contact test-user added!" in response.output
assert "{'mobile': '0'}" in response.output

The rest of the tests will pretty much follow the same format.

Testing the list command

def test_list():
    response = runner.invoke(list)
    assert response.exit_code == 0
    assert "Here's a list of all your contacts:" in response.output
    assert "'test-user': {'mobile': '0'}" in response.output

The list command doesn’t take any arguments or options so we just call it directly: response = runner.invoke(list).

Testing the view command

def test_view():
    response = runner.invoke(view, "test-user")
    assert response.exit_code == 0
    assert "{'mobile': '0'}" in response.output

Testing the update command

def test_update():
    response = runner.invoke(update, ["test-user", "-m", "12345"])
    assert response.exit_code == 0
    assert "Contact updated!" in response.output
    assert "{'mobile': '12345'}" in response.output

Testing the delete command

def test_delete():
    response = runner.invoke(delete, "test-user")
    assert response.exit_code == 0
    assert "Contact deleted!" in response.output

    # call view on test-user to confirm it doesn't exist
    response = runner.invoke(view, "test-user")
    assert response.exit_code == 0
    assert "The contact you searched for doesn't exist" in response.output

Improvements

As your application grows, you may want to consider using fixtures and set up things like runner in a conftest.py file. We got away with it here because our tests were simple and all in a single file. Once multiple test files are introduced, following the approach we used here would lead to a lot of unnecessarily duplicated code.

Also, we made direct calls to our API in the tests. This operation should ideally be mocked.

What we’ll create

A contacts app with the following functionality:

# add a new contact with name=Peter and mobile=01034512
$ contacts add Peter --mobile 01034512
Contact Peter added!
{ 'mobile': '01034512' }

# view all contacts
$ contacts list
Here's a list of all your contacts:
{ ...  # list of all contacts }

# view single contact
$ contacts view Peter
{ 'mobile': '01034512' }

$ contacts view Meg
The contact you searched for doesn't exist

# update mobile number for Peter
$ contacts update Peter --mobile 00000
Contact updated!
{ 'mobile': '00000' }

# delete the contact Peter
$ contacts delete Peter
Contact deleted!

This post is a bit long. If you just want to take a look at the code or see how the app works, check out the repo.

Setting up Firebase

As mentioned earlier, Firebase will be used to store our data. To set up the database, sign in on Firebase using your google account.

After signing in successfully, click on the Go to console button on the top right. There, you can create a new project. Follow the required steps to create a new project.

Once the project is created, you’ll see a Develop sidebar on the left. Click on Database to set up the database. Cloud Firestore will be displayed as the first option but that’s not what we’ll be using. Scroll down and you’ll see a Realtime Database option. Click on Create database. Next, you’ll be presented with the security options.

Since we’ll not be handling authentication, choose Start in test mode for the security rules. Your database is now created 🎉

Now for the final step in setting up the database, click on the little plus sign + to add a new child node called contacts. This will hold our contacts (obviously 😅). Also, add a test contact just to see how the database will be structured.

The database should now look like this:

App setup

We’ve already gone through the basics of setting up in previous posts so I won’t go into much detail on that. Here are the steps:

$ mkdir contacts-cli

$ cd contacts-cli

$ virtualenv env

$ . env/bin/activate

(env) $ mkdir app

(env) $ touch app/__init__.py

(env) $ touch setup.py

(env) $ touch .env

.env

This is where the api endpoint url will be stored. Firebase provides a REST API for the database so that’s what we’ll use. If you already followed the steps and created the database, just grab the link provided at the top of the database and set it in your .env file.

export URL=https://[YOUR_PROJECT_ID].firebaseio.com

Make sure to run . .env to load your environment variables, otherwise URL will be None.

setup.py

To learn more about the setup file, you can go back to this post.

from setuptools import setup, find_packages

setup(
        name="contacts",
        py_modules=find_packages(),
        install_requires=[
            "Click",
            "requests",
            ],
        entry_points="""
        [console_scripts]
        contacts=app:cli
        """,
        )

Then run pip install --editable . in your terminal to install the app.

1. app/__init__.py

import click

@click.group(invoke_without_command=True)
@click.pass_context
def cli(ctx):
    if ctx.invoked_subcommand is None:
        click.echo("Welcome to the contacts app 🥳")
        click.echo("Run contacts --help for options.")

We need to use @click.group to allow us to include sub-commands.

If the main command is invoked without any sub-command, a welcome message is shown. How does the main command know if a sub-command has been passed in or not? By looking at the context. if ctx.invoked_subcommand is None, then no sub-command has been passed - show the welcome message.

Everything is set up now, so we can start working on the functionality.

Creating contacts

import os
import requests
import click

...

@cli.command()
@click.argument('name')
@click.option('--mobile', '-m', required=True)
def add(name, mobile):
    """
    Add a new contact
    """
    response = requests.put('{}/contacts/{}.json'
                            .format(os.getenv('URL'), name),
                            json={'mobile': '{}'.format(mobile)})
    click.echo('Contact {} added!'.format(name))
    click.echo(response.json())

You’ll notice that I used PUT for the request instead of POST. I did this to avoid generating unique IDs for my data on Firebase. Feel free to use POST instead and see how you can work with the generated UIDs.

The add command takes two parameters - a name argument, and a mobile option. Both are required. Running contacts add --help shows us what options we have.

(env) $ contacts add --help

Usage: contacts add [OPTIONS] NAME

  Add a new contact

Options:
  -m, --mobile TEXT  [required]
  --help             Show this message and exit.

Let’s add a new contact:

(env) $ contacts add Chris -m 34243234
Contact Chris added!
{'mobile': '34243234'}

Getting all contacts

...

@cli.command()
def list():
    """
    View all contacts
    """
    response = requests.get('{}/contacts.json'
                            .format(os.getenv('URL')))
    click.echo('Here's a list of all your contacts:')
    click.echo(response.json())

The list command simply does a GET request on the contacts node and returns the result.

(env) $ contacts list
Here's a list of all your contacts:
{'Chris': {'mobile': '34243234'}, 'Peter': {'mobile': '8487344'}}

Getting one contact

...

@cli.command()
@click.argument('name')
def view(name):
    """
    View single contact
    """
    response = requests.get('{}/contacts/{}.json'
                            .format(os.getenv('URL'), name))
    if not response.json():
        click.echo("The contact you searched for doesn't exist")
    else:
        click.echo(response.json())

The view command is similar to list, except it takes a name argument and adds it to the url to return the contact for that name only. If the name doesn’t exist in the database, null is returned. We take advantage of that in the if block to return the appropriate response.

(env) $ contacts view Peter
{'mobile': '8487344'}

(env) $ contacts view Meg
The contact you searched for doesn't exist

Updating a contact

...

@cli.command()
@click.argument('name')
@click.option('--mobile', '-m', required=True)
def update(name, mobile):
    """
    Update contact
    """
    response = requests.patch('{}/contacts/{}.json'
                              .format(os.getenv('URL'), name),
                              json={'mobile': '{}'.format(mobile)})
    click.echo('Contact updated!')
    click.echo(response.json())

This works exactly the same as the create command, except we’re using PATCH instead of PUT (which apparently, doesn’t make any difference here. Making a PATCH on a non-existent node creates it, just like PUT would do. Try it out.)

(env) $ contacts update Peter -m 552
Contact updated!
{'mobile': '552'}

And finally, deleting a contact

Similar to the view command. The only change we’ll make is using DELETE instead of GET.

...

@cli.command()
@click.argument('name')
def delete(name):
    """
    Delete contact
    """
    requests.delete('{}/contacts/{}.json'
                    .format(os.getenv('URL'), name))
    click.echo('Contact deleted!')

null is returned on successful deletion so we don’t need to return response. The request is called directly, and a success message is shown afterwards. This can be done better but it’ll do for now.

(env) $ contacts delete Peter
Contact deleted!

I meant to cover exporting (writing) the data to a local JSON file or sqlite database, but this post is already too long. These are interesting things to try if you want to learn more.

In the next post, we’ll add tests to the app using pytest.

# helloworld.py

import click

@click.command()
@click.option('-c', '--case',
              type=click.Choice(['upper', 'lower']),
              prompt=True)
@click.argument('person', default='you')
def hello(case, person):
    response = "Hello World! Also, hey {} ☺️".format(person)
    if case == 'upper':
        click.echo(response.upper())
    elif case == 'lower':
        click.echo(response.lower())
    else:
        click.echo(response)

Save and run the app:

(venv) $ hello
Case (upper, lower): upper
HELLO WORLD! ALSO, HEY YOU ☺️

(venv) $ hello sally
Case (upper, lower): lower
hello world! also, hey sally ☺️

You may also set a custom prompt string if you wish:

# helloworld.py

import click

@click.command()
@click.option('-c', '--case',
              type=click.Choice(['upper', 'lower']),
              prompt='Please enter case')
@click.argument('person', default='you')
def hello(case, person):
    response = "Hello World! Also, hey {} ☺️".format(person)
    if case == 'upper':
        click.echo(response.upper())
    elif case == 'lower':
        click.echo(response.lower())
    else:
        click.echo(response)

The set string will now be displayed instead of the default prompt:

(venv) $ hello
Please enter case (upper, lower): upper
HELLO WORLD! ALSO, HEY YOU ☺️

Let’s edit our code a bit to see how we can integrate arguments.

# helloworld.py

import click

@click.command()
@click.option('-c', '--case', type=click.Choice(['upper', 'lower']))
@click.argument('person', default='you')
def hello(case, person):
    response = "Hello World! Also, hey {} ☺️".format(person)
    if case == 'upper':
        click.echo(response.upper())
    elif case == 'lower':
        click.echo(response.lower())
    else:
        click.echo(response)

Just like with commands and options, Click provides a decorator to add arguments.

@click.argument('person', default='you')

We specify that hello() should expect an argument person to be passed in when it’s called, and add a default value so that it’s ok to call the command without passing in the argument.

Saving and running the app gives the following results:

(venv) $ hello  # no argument - default will be used
Hello World! Also, hey you ☺️

(venv) $ hello Meg  # Meg is the argument
Hello World! Also, hey Meg ☺️

(venv) $ hello chris -c upper   # combining arguments with options
HELLO WORLD! ALSO, HEY CHRIS ☺️

With options and arguments, you can add a lot of functionality to a single command.

Options

Options are used to alter the default behavior of commands. This is often accomplished by passing in parameters in the form of --option or -o for short.

Let’s assume that instead of just printing out "Hello World!", we wanted to have the option of printing it in all lowercase characters or all uppercase characters. We can add this functionality by adding an option() decorator.

# helloworld.py

import click

@click.command()
@click.option('-c', '--case', type=click.Choice(['upper', 'lower']))
def hello(case):
    if case:
        click.echo('You selected {} case.'.format(case))
    # click.echo('Hello World!')

Then run the app, passing in the created option:

(venv) $ hello -c upper  # --case would also work
You selected upper case.

(venv) $ hello -c lower
You selected lower case.

Now try passing in some other option, like "normal":

(venv) $ hello -c normal
Usage: hello [OPTIONS]
Try "hello --help" for help.

Error: Invalid value for "-c" / "--case": invalid choice: normal. (choose from upper, lower)

It gives us an error: invalid choice: normal and even guides us on the options to choose from (choose from upper, lower), since that’s what we specified type=click.Choice(['upper', 'lower']). Error handling on that part is handled by default.

Ok, now that the option is passed in successfully, we can add in the logic to see our desired output.

import click

@click.command()
@click.option('-c', '--case', type=click.Choice(['upper', 'lower']))
def hello(case):
    response = "Hello World!"
    if case == 'upper':
        click.echo(response.upper())
    elif case == 'lower':
        click.echo(response.lower())
    else:
        click.echo(response)

All we’ve done is add an if-else to check the option passed in, and return the text in either upper case, lower case, or just normal. Now when we install our app again and run it, we get these results:

(venv) $ hello
Hello World!

(venv) $ hello -c upper
HELLO WORLD!

(venv) $ hello -c lower
hello world!

In the next post, we’ll enhance our command even more by adding arguments.

(venv) $ hello
Hello World!

That’s where setuptools comes in. Setuptools helps us bundle our script so we can install it and run it like an app ($ hello), instead of like a script ($ python hello-world.py).

Setuptools is a collection of enhancements to the Python distutils that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages. - Docs

The first thing we’ll need to do is add a new file to our project and call it setup.py. We’ll also rename hello-world.py to helloworld.py (remove the dash). I’ll explain why this is necessary when we get to the setup() section. Our project structure should now look like this:

hello-world-cli
    |- helloworld.py  # renamed from hello-world.py
    |- setup.py  # new file

In the helloworld.py, remove the if __name__ == '__main__': section. The code should now look like this:

import click

@click.command()
def hello():
    click.echo('Hello World!')

In the setup.py file, add this code:

from setuptools import setup

setup(
    name='hello-world-cli',
    py_modules=['helloworld'],
    install_requires=[
        'Click',
    ],
    entry_points='''
        [console_scripts]
        hello=helloworld:hello
    ''',
)

Let’s examine what each line of code does.

from setuptools import setup

We need to import setup from setuptools in order to use it.

Next, we call the setup() function and pass in a number of parameters:

name='hello-world-cli'

This’ll be the name of our CLI app.

py_modules=['helloworld']

This tells the setup() where to find our main module to execute. Our main file here is helloworld.py so we set helloworld as the module. We can keep it that simple since we only have one file for now, but as our app grows we’ll replace py_modules with packages=find_packages(). More on that later.

install_requires=['Click']

Our app has Click as a dependency, so it needs to be installed for it to work. setup() needs to know about this so it can include it as it packages our project. install_requires accepts a list of values, so if we had more dependencies, we would list all of them here.

entry_points='''
          [console_scripts]
          hello=helloworld:hello
      '''

This is the part that saves us the hassle of having to run the app with python ... every time, and enables us to just execute hello to run it.

It’s important to note that if we maintained our previous name (hello-world.py), the setup would have failed at entry_points. This is because python module names shouldn’t have dashes.

We can now install our app in our environment and run it. To install it, run:

(venv) $ pip install --editable .

Then execute the created command:

(venv) $ hello
Hello World!

Python 3.x

Getting started with Python is easy. Most *nix systems come with Python pre-installed. We’ll be using Python 3. To check which version of Python you have, open up your terminal and run python -V. You should get an output similar to this:

Python 2.7.10

This means the default Python version installed on my computer is 2.7.10. Most Macs ship with Python 2 by default. If you got a 3.x.x, then you’re good to go. After installing Python 3, running python3 -V outputs:

Python 3.7.3

If you don’t have Python installed, you can head over to python.org, download the appropriate installer for your system, and run it. This step is necessary if you’re on Windows.

pip

pip is a package manager for Python. It allows us to install libraries and dependencies that we’ll need to use for our project. It comes pre-installed with Python. To check if you have it, you can run pip -V or pip3 -V. You should see something like:

pip 19.0.3 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)

Virtualenv

When working on your Python projects, it’s always a good idea to create isolated environments to work in. This is because different projects will require different dependencies or different versions of those dependencies. Installing these globally will be hard to manage and will just mess up your system. Installing them in a project’s environment keeps things clean. There’s a couple of options to deal with environments in Python but we’ll focus on virtualenv.

We’re going to create a new directory and create a virtual environment in it. On your terminal, run:

$ mkdir hello-world-cli && cd hello-world-cli

The command above creates a new directory called hello-world-cli and enters that directory.

Next, we’ll install virtualenv using pip and create our virtual environment.

$ pip3 install virtualenv

$ virtualenv venv

We now have a Python virtual environment in our directory. To use it, we need to activate it by running:

$ source venv/bin/activate
(venv) $

You’ll notice the change in your terminal prompt when the environment is activated. The environment name is prefixed at the prompt. That’s how you know you’re working in the environment.

Click

Now that we have our environment ready, we can start working on our project. Python provides a number of packages to help in creating CLI apps. We’ll be using Click, so we need to install it to get started.

(venv) $ pip install Click

That’s all we need to get started.

Hello World!

As usual, we’ll start off with a "Hello World!" program.

Still in the hello-world-cli directory, create a file called hello-world.py:

(venv) $ touch hello-world.py

Open the file in your preferred text editor and add this code:

import click

@click.command()
def hello():
    click.echo('Hello World!')

if __name__ == '__main__':
    hello()

Let’s examine what’s happening at each line.

import click

Here, we import click so we can use it to create our commands.

@click.command()

Whenever you see something on top of a function with an @ symbol in Python, that’s a decorator. Without going into a lot of details on decorators, it’s enough to understand (for now) that they simply modify the behavior of the functions they "decorate". Click uses the concept of decorators to convert Python functions into commands that can be directly executed through the terminal. The decorator here converts hello() into a command. We’ll learn more about commands later.

def hello():

This is how we define functions in Python.

click.echo('Hello World!')

Calling hello() will run click.echo('Hello World!') which displays the text "Hello World!" on the terminal.

if __name__ == '__main__':

This is the main entry point of our script.

hello()

This invokes our function/command.

Ok, time to see the results. Save the file, go back to the terminal, and run the program:

(venv) $ python hello-world.py
Hello World!

For example:

[0, 1, 1, 2, 3, 5, 8, 13, 21...]

We can represent this in the formula:

fib(n) = fib(n-1)+fib(n-2)

With this formula, we can write a simple recursive function to solve for fib(n).

Note: Only use this to test for small numbers, preferably n < 10. I’ll explain later

def fib(n):
    if n < 2:  # base case
        return n
    return fib(n-1) + fib(n-2)

if __name__ == "__main__":
  print(fib(7))

Why do we need a base case?

It’s easy enough to convert the formula directly into

def fib(n):
    return fib(n-1) + fib(n-2)

The problem with this though is that when you run it, it throws a RecursionError. This simply means that there’s possibly an infinite loop in the program. A base case in a recursive function tells the function when to stop (to avoid going into an infinite loop) and is usually something that is already known or that can be solved for easily without needing the function. In our case here, we know from the definition that any number in the sequence, except for the first and second, is the sum of the previous two. We use that to form our base case if n < 2: return n.

Making it more efficient

Remember when I told you to only test the program with small values of n? Here’s why.

As it stands, every call to fib() results in two more calls to fib() in the return statement. The call tree grows exponentially. 15 calls are required to compute fib(5), 177 calls for fib(10), 21,891 for fib(20)… you get the point. To solve this problem, we can use memoization.

Memoization helps avoid unnecessary calculation of the same values if they were already previously calculated. It works just like memorization for humans. You already have 2 x 2 memorized and can give the answer immediately without having to use a calculator. 799 x 377? You probably need to use a calculator for that. If, for some reason, you find that you get asked 799 x 377 a lot, it would be nice to have it memorized so you don’t have to calculate it every other time. The value of 799 x 377 will always remain the same, so all you have to do is calculate it once, save the value in your "cache" (memory), and retrieve it every time you need it.

Luckily, python has a built-in decorator that does just that.

from functools import lru_cache

@lru_cache(maxsize=None)
def fib(n):
    if n < 2:  # base case
        return n
    return fib(n-1) + fib(n-2)

if __name__ == "__main__":
  print(fib(50))

Now you can test with large numbers safely.

Installing the requirements

Create a virtual environment and install Django.

$ virtualenv venv # creates a virtual environment
$ source venv/bin/activate # activates the virtual environment
(venv)$ pip install Django

That’s all we’ll need for now.

Creating the project

Use the Django startproject command to create the project we’ll be working on.

$ django-admin startproject todoapi

This creates a todoapi directory with generated files to start us off. The directory should look something like this:

manage.py
todoapi/
    __init__.py
    settings.py
    urls.py
    wsgi.py

Database setup

To keep things simple, we’ll use the SQlite database, so we don’t have to install anything since it comes with Python. In addition, that’s already setup for us in todoapi/settings.py. Have a look at the code and you should see something similar to this:

# Database
# https://docs.djangoproject.com/en/2.1/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

Creating models

To create models for our database, we first need to create a todos "app". This is where we’ll implement our features.

$ python manage.py startapp todos

This will create a todos directory with the following files:

migrations/
    __init__.py
__init__.py
admin.py
apps.py
models.py
tests.py
views.py

This app should be added to the todoapi/settings.py:

INSTALLED_APPS = [
    ...,
    'todos'
]

Ok, now that Django knows about our app, open the models.py file to add the models. We’ll only create one model called Todo:

from django.db import models


# Create your models here.
class Todo(models.Model):
    todo = models.CharField(max_length=100, null=False,
                            help_text="This field is required")
    done = models.BooleanField(default=False)

This model will create a table for us with todo and done columns to save our data. The todo column will save the actual todo, and the done column will save the status – whether it’s done or not. Notice that it is a BooleanField, meaning we’ll save True when done or False when not done, which is the default.

Now all we need to do is run makemigrations to notify Django of our models, and migrate to do the actual migration.

$ python manage.py makemigrations todos
$ python manage.py migrate

Creating the endpoints

We’ll have two endpoints for now:

• /todos to GET a list of all our todos
• /todos/<id> to GET the todo matching the id

To test this out, we’ll create two simple views in the todos/view.py file.

from django.http import JsonResponse
from django.shortcuts import get_object_or_404

from .models import Todo

# Create your views here.
def todos(request):
    todos_list = Todo.objects.all()
    data = {
        "todos": list(todos_list.values(
            "todo", "done"
        ))
    }
    return JsonResponse(data)

def single_todo(request, pk):
    todo = get_object_or_404(Todo, pk=pk)
    data = {
        "todo": todo.todo,
        "done": todo.done
    }
    return JsonResponse(data)

Create a new file called urls.py in your todos app and add the following code:

# todos/urls.py
from django.urls import path

from .views import todos, single_todo

urlpatterns = [
    path("todos/", todos, name="todos"),
    path("todos/<int:pk>", single_todo, name="single_todo")
]

Go to todoapi/urls.py and include the todos url you just created:

# todoapi/urls.py
from django.contrib import admin
from django.urls import path, include  # add include here

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include('todos.urls'))  # add this
]

Set up your admin and test the endpoints

Create a superuser:

$ python manage.py createsuperuser

Next, register your models in the /todos/admin.py file so you can have access to them from the admin dashboard:

from django.contrib import admin
from .models import Todo

# Register your models here.
admin.site.register(Todo)

That’s it! Now run the server and go to http://127.0.0.1:8000/admin. Log in with the details you registered for the superuser above to add your todos.

You can retrieve the todos created via a rest client via the endpoints

• http://127.0.0.1:8000/todos
• http://127.0.0.1:8000/todos/<id>

"When health is absent, wisdom cannot reveal itself, art cannot manifest, strength cannot fight, wealth becomes useless, and intelligence cannot be applied." ― Herophilus

Let’s face it. As developers, it’s easy to let things get a bit out of hand when it comes to health. Many times we find ourselves in front of screens for hours on end. We sleep really late (or not at all) sometimes. Unless you’re disciplined enough to follow a schedule, you might find yourself not having time to eat proper meals so snacking becomes a habit. All these are usually sacrifices made to fix some bug or find the most efficient solution to some problem. Usually, our health suffers as a result. While this might seem to work out just fine for the moment, we don’t look at the bigger picture - how it will affect us in the future.

I think, if healthier practices were adopted, developers would be much more productive and would be able to accomplish some tasks in less time simply because the brain is much clearer. I could cite surveys and research here to prove this point, but simple personal experience will suffice. It’s hard enough to get anything done when you’re down with a cold. There is an intimate connection between the mind and the body, and when one suffers, the other no doubt feels it. Below are some points to watch out for if you’re looking to keep fit physically and mentally.

Watch what you eat 🥘

Just like your car wouldn’t run without the correct fuel, our bodies also need the right fuel to work efficiently. Make sure you find time to eat a properly balanced meal - preferably a big breakfast to give you enough energy for the day, a good lunch, and a light supper. This is, of course, assuming a regular daytime 9 to 5 type of schedule. When working odd hours, you can adjust accordingly.

Have some exercise 🏃🏽‍♂️

I won’t go as far as some have to say that sitting kills 😄 but it’s definitely a good idea to wake up and stretch at least once or twice every hour. The Pomodoro technique is great for this because you can take your breaks to get off your chair for five minutes. I have found that taking walks is a great way to refresh your mind especially when you’re struggling with something. You can also take advantage of the fresh air of early morning jogs.

Stay hydrated 💧

If you wait until you feel thirsty, you’ve waited too long.

Take a break 🚶

We’ve all been there. Sometimes our minds work best when we step away from it all and take a break. Again, taking walks work great! I took a course on Coursera some time back called "Learning How to Learn" that explains the science behind how this works. You can check it out.

Get enough sleep 🛌💤

According to this study, sleep helps clear the brain of toxins that build up while we are awake. Depriving yourself of sleep only builds up these toxins and your brain gets all foggy. This only leads to spending more hours trying to figure something out, which leads to more sleep deprivation, and it becomes a vicious cycle. Everyone needs different amounts of sleep but if you can clock in 7 or 8 hours, that should be enough. Again, this assumes a regular daytime 9 to 5 type of schedule. I understand that some people find it easier to work late at night and that’s cool. I’m a morning person myself so I’d rather sleep early and rise early.

I might get into more details on each of these steps in future posts in the series. Topics of interest include taking a closer look at coffee and energy drinks to see if they really make us more productive.

function someFunction() {
    // some code here
}

function anotherFunction() {
    // some more code here
}

someFunction and anotherFunction are two different code blocks. These two different code blocks could also be considered as two different scopes. What that means is that the variables declared in someFunction only affect that block of code, and those declared in anotherFunction only affect that block of code. They are "scoped" in that sense.

To illustrate:

var name = "Kenny"

function someFunction() {
    var name = "Kyle"
    console.log("someFunction block:", name) // someFunction block: Kyle
}

function anotherFunction() {
    var name = "Timmy"
    console.log("anotherFunction block:", name) // anotherFunction block: Timmy
}

console.log("global", name) // global Kenny

As you can see, the same variable name retains its global value although it gets redefined within someFunction and anotherFunction.

Now here is where it can get a bit tricky. With that knowledge in mind, it is easy to assume that this would always be the case whenever we have code blocks. Except, it doesn’t work the same with if/else statements and for loops.

var name = "Kenny";

if (name) {
  var name = "Kyle";
  console.log("if block:", name); // if block: Kyle
}

console.log("global", name); // global Kyle

The name variable inside the if block resets the value of name. The same happens in a for loop:

var name = "Kenny";

for (var i = 0; i < 1; i++) {
  var name = "Kyle";
  console.log("for loop block:", name); // for loop block: Kyle
}

console.log("global", name); // global Kyle

The solution is to use the let keyword instead of var.

var name = "Kenny";

if (name) {
  let name = "Kyle";
  console.log("if block:", name); // if block: Kyle
}

console.log("global", name); // global Kenny

var name = "Kenny";

for (var i = 0; i < 1; i++) {
  let name = "Kyle";
  console.log("for loop block:", name); // for loop block: Kyle
}

console.log("global", name); // global Kenny

Working with constants

Logarithm to base e

Math.E; // 2.718281828459045

Logarithm to base 10

Math.LN10; // 2.302585092994046

Logarithm to base 2

Math.LN2; // 0.6931471805599453

Base 10 logarithm of e

Math.LOG10E; // 0.4342944819032518

Base 2 logarithm of e

Math.LOG2E; // 1.4426950408889634

🥧

Math.PI; // 3.141592653589793

Square root of 1/2

Math.SQRT1_2; // 0.7071067811865476

Square root of 2

Math.SQRT2; // 1.4142135623730951

Infinity

Infinity; // Infinity

Rounding

Math.round returns the value of a number rounded to the nearest integer.

Math.round(4.2); // 4
Math.round(4.7); // 5
Math.round(4.5); // 5. Half-way values are always rounded up

Speaking of rounding up, Math.ceil():

Math.ceil(4.2); // 5
Math.ceil(4.7); // 5
Math.ceil(-4.7); // -4. Ceiling a negative number will round towards zero

Math.floor() rounds down:

Math.floor(4.2); // 4
Math.floor(4.7); // 4
Math.floor(-4.7); // -5. Flooring a negative number will round away from zero

Modulus (%)

Returns the remainder after (integer) division.

42 % 10; // 2
-40 % 10; // -0 🤔

Trigonometry

Sine

Math.sin(60); // -0.3048106211022167

Cosine

Math.cos(60); // -0.9524129804151563

Tangent

Math.tan(60); // 0.320040389379563

Incrementing (++)

++ increments its operand by 1.

// postfix: returns the value before incrementing
let a = 4, // 4
  b = a++, // 4
  c = a; //5

// prefix: returns the value after incrementing
let a = 4, // 4
  b = ++a, // 5
  c = a; //5

Decrementing (–)

-- decrements its operand by 1.

// postfix: returns the value before decrementing
let a = 4, // 4
  b = a--, // 4
  c = a; //3

// prefix: returns the value after decrementing
let a = 4, // 4
  b = --a, // 3
  c = a; //3

Exponentiation (**)

// Math.pow() or ** can be used
let a = 4,
  b = 2,
  c = Math.pow(a, b), // 16
  d = a ** b; // 16

Getting maximum and minimum

Math.max(4.2, 4.7); // 4.7
Math.min(4.2, 4.7); // 4.2

Getting maximum and minimum from an array:

const arr = [1, 2, 3, 4, 5, 6, 7, 8, 9],
  max = Math.max(...arr), // 9
  min = Math.min(...arr); // 1

Getting roots √

Square Root

Math.sqrt(16); // 4

Cube Root

Math.cbrt(27); // 3

To find the nth-root, use the Math.pow() function and pass in a fractional exponent.

// This finds the 6th root of 64
Math.pow(64, 1 / 6); // 4

Much more complex calculations can be done by combining one or more of these operations.

Modules are single Python files that can be imported. Any python file can be a module. For example, if I have two Python files: module.py and hello.py in the same directory:

# module.py

def hello(name):
    print("Hello {}".format(name))

I can import that module in my hello.py:

#hello.py

import module

module.hello("World!") # Hello World!

The same can be done in the interpreter:

>>> from module import hello
>>> hello("World!") # Hello World!

Packages

Packages are made up of multiple Python files (or modules), and can even include libraries written in different languages like C or C++. Seeing an __init__.py file in a folder typically tells you that that folder is a Python package. The __init__.py doesn’t have to contain any code – sometimes it does – it just has to be there for Python take that particular folder as a package.

📁 my_package
    |- __init__.py
    |- module.py

# __init__.py

from my_package.module import hello

When you import my_package in your script, the __init__.py script will be run, giving you access to all of the functions in the package. In this case, it only gives access to the module.hello function.

Setting up the HTML

<!-- this will fetch the data -->
<button onclick="loadData()">Load data</button>

<!-- this will show our spinner -->
<div hidden id="spinner"></div>

The spinner visibility is hidden by default using the built-in hidden attribute. This is the element we’ll manipulate in order to show and hide it as desired.

Creating the CSS spinner

#spinner:not([hidden]) {
  position: fixed;
  top: 0;
  left: 0;
  right: 0;
  bottom: 0;
  display: flex;
  justify-content: center;
  align-items: center;
}

#spinner::after {
  content: "";
  width: 80px;
  height: 80px;
  border: 2px solid #f3f3f3;
  border-top: 3px solid #f25a41;
  border-radius: 100%;
  will-change: transform;
  animation: spin 1s infinite linear;
}

@keyframes spin {
  from {
    transform: rotate(0deg);
  }
  to {
    transform: rotate(360deg);
  }
}

Getting things working with Javascript

const spinner = document.getElementById("spinner");

function loadData() {
  spinner.removeAttribute("hidden");
  fetch("https://www.mocky.io/v2/5185415ba171ea3a00704eed?mocky-delay=5000ms")
    .then((response) => response.json())
    .then((data) => {
      spinner.setAttribute("hidden", "");
      console.log(data);
    });
}

When the loadData function is called, it removes the hidden attribute (spinner.removeAttribute('hidden')). I’ve set the delay to 5 seconds (mocky-delay=5000ms) so we can see the spinner in action. After that, the hidden attribute is restored (spinner.setAttribute('hidden', '')), hiding the spinner. Ideally, this would happen after the data being fetched is returned.

Here’s the pen

There are a couple of ways to do this.

The simplest is to use the @pytest.mark.skip decorator like so:

import pytest

def test_stuff(): # this test will be executed
    pass

@pytest.mark.skip(reason="just testing if skip works")
def test_other_stuff(): # this one will be skipped
    pass

Running your tests should produce the output below:

collected 2 items
skip.py .s                           [100%]

===================== 1 passed, 1 skipped in 0.05 seconds ===============

Notice the little s. It shows the test that was skipped. Pytest also tells us that 1 passed, 1 skipped. If you need a more verbose output, you can use the -rs flag pytest skip.py -rs:

collected 2 items
skip.py .s                    [100%]
==================== short test summary info ===============
SKIP [1] skip.py:14: just testing if skip works

=================== 1 passed, 1 skipped in 0.02 seconds ==========

The test above was skipped even before it started. This isn’t always ideal. You can have more control over how the test is skipped by using the pytest.skip(reason) function:

import pytest

def setup_stuff():
    return False

def test_stuff(): # this test will be executed
    pass

def test_other_stuff(): # this one will be skipped if setup_stuff() returns false
    if not setup_stuff():
        pytest.skip("setup failed")
    else:
        pass

collected 2 items
skip.py .s                  [100%]
=================== short test summary info =============================
SKIP [1] skip.py:12: setup failed

==================== 1 passed, 1 skipped in 0.05 seconds =================

If you prefer to check that the condition is satisfied before the test starts, then you can use skipif:

import pytest

def setup_stuff():
    return False

def test_stuff(): # this test will be executed
    pass

@pytest.mark.skipif(not setup_stuff(), reason="setup failed")
def test_other_stuff(): # this one will be skipped if setup_stuff() returns false
    pass

collected 2 items
skip.py .s                  [100%]
=================== short test summary info =============================
SKIP [1] skip.py:12: setup failed

==================== 1 passed, 1 skipped in 0.05 seconds =================

There are many other ways you can customize your tests to skip depending on certain conditions as explained in the docs.

Let’s demonstrate with a simple example:

def add_numbers(num1, num2):
    print("Add function started...")
    result = num1 + num2
    print("Numbers added. Returning result...")
    return result

def test_add_numbers():
    assert add_numbers(2, 3) == 3

Running the test above fails with the given traceback:

collected 1 item
print.py F                 [100%]

============================ FAILURES ============================

    def test_add_numbers():
>       assert add_numbers(2, 3) == 3
E       assert 5 == 3
E        +  where 5 = add_numbers(2, 3)

print.py:9: AssertionError
---------------------- Captured stdout call ----------------------------
Add function started...
Numbers added. Returning result...
===================== 1 failed in 0.11 seconds ====================

Notice the Captured stdout call section. It captured out two print statements. During test execution, any output sent to stdout and stderr is captured. By default, the captured output is only displayed if the test fails. Changing our assert add_numbers(2, 3) == 3 to assert add_numbers(2, 3) == 5 and running the test again gives the following output:

collected 1 item
print.py .                       [100%]

========================= 1 passed in 0.05 seconds ===========================

You can access your captured output in your tests by using readouterr():

def add_numbers(num1, num2):
    print("Add function started...")
    result = num1 + num2
    print("Numbers added. Returning result...")
    return result

def test_add_numbers(capsys):
    result = add_numbers(2, 3)
    captured = capsys.readouterr()
    assert "Hello" in captured.out
    assert result == 3

collected 1 item
print.py F                             [100%]

====================== FAILURES =======================

capsys = <_pytest.capture.CaptureFixture object at 0x10ba25d30>

    def test_add_numbers(capsys):
        result = add_numbers(2, 3)
        captured = capsys.readouterr()
>       assert "Hello" in captured.out
E       AssertionError: assert 'Hello' in 'Add function started...\nNumbers added. Returning result...\n'
E        +  where 'Add function started...\nNumbers added. Returning result...\n' = CaptureResult(out='Add function started...\nNumbers added. Returning result...\n', err='').out

print.py:11: AssertionError
======================== 1 failed in 0.09 seconds =======================

It’s important to note that each readouterr() call snapshots the output so far, so we can call it repeatedly, checking the output of our test step by step.

def add_numbers(num1, num2):
    print("Add function started...")
    result = num1 + num2
    print("Numbers added. Returning result...")
    return result

def test_add_numbers(capsys):
    result = add_numbers(2, 3)
    captured = capsys.readouterr()
    assert "Add function started" in captured.out
    assert "Numbers added. Returning result" in captured.out
    # thus far, readouterr() out only has the two print statements from the add_numbers function
    assert result == 5
    print("thank you, next")
    captured = capsys.readouterr()
    # here, readouterr() out doesn't have the two outputs from above
    # it now contains the new value, "thank you, next"
    assert captured.out == "thank you, next\n"

collected 1 item
print.py .                       [100%]

========================= 1 passed in 0.05 seconds ===========================

Here’s an example:

# login.py

def check_email_format(email):
    """check that the entered email format is correct"""
    pass

def test_email_exception():
    """test that exception is raised for invalid emails"""
    pass

This is probably something you want to do if you’re implementing a system with email authentication. The example is oversimplified, but it serves the purpose of this post well.

To test for raised exceptions, pytest offers a handy method: pytest.raises. Let’s see how to use it in our example:

import re
import pytest

def check_email_format(email):
    """check that the entered email format is correct"""
    if not re.match(r"(^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$)", email):
        raise Exception("Invalid email format")
    else:
        return "Email format is ok"

def test_email_exception():
    """test that exception is raised for invalid emails"""
    with pytest.raises(Exception):
        assert check_email_format("good@email.com")

The check_email_format method takes in an email and checks that it matches the regex pattern given. If it does, it returns "Email format is ok", otherwise, an exception is raised.

Using pytest.raises in a with block as a context manager, we can check that an exception is actually raised if an invalid email is given. Running the tests on the code as it is above should fail:

collected 1 item
login.py F                [100%]

==================== FAILURES ========================

    def test_email_exception():
        """test that exception is raised for invalid emails"""
        with pytest.raises(Exception):
>           assert check_email_format("good@email.com")
E           Failed: DID NOT RAISE <class 'Exception'>

login.py:16: Failed

Notice it says Failed: DID NOT RAISE <class 'Exception'>. If an exception is not raised, the test fails. I found this particularly useful. We passed in a valid email format (according to our standards here) so the test works as expected. Now we can make it pass.

import re
import pytest

def check_email_format(email):
    """check that the entered email format is correct"""
    if not re.match(r"(^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$)", email):
        raise Exception("Invalid email format")
    else:
        return "Email format is ok"

def test_email_exception():
    """test that exception is raised for invalid emails"""
    with pytest.raises(Exception):
        assert check_email_format("bademail.com") # invalid email format to raise exception

Run your test: pytest login.py:

collected 1 item

login.py .              [100%]

====================== 1 passed in 0.05 seconds ======================

You can also add an extra check for the exception message:

import re
import pytest

def check_email_format(email):
    """check that the entered email format is correct"""
    if not re.match(r"(^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$)", email):
        raise Exception("Invalid email format")
    else:
        return "Email format is ok"

def test_email_exception():
    """test that exception is raised for invalid emails"""
    with pytest.raises(Exception) as e:
        assert check_email_format("bademail.com")
    assert str(e.value) == "Invalid email format"

In this series, I’ll be exploring the cool features that pytest offers to help make testing Python code a breeze.

Preparing your python environment

Let’s set up our testing environment with virtualenv. We’ll be working with Python 3. cd into your working directory and create a new virtual environment:

python3 -m venv env

Activate the virtual environment:

source env/bin/activate

Installing pytest

pip install pytest

That’s all you need to do to get pytest ready. You can check the installed version by running:

pytest --version

Creating your first test

Say we need to create an app that says hello to the name you give it when it runs.

# simple hello world test

def hello_world(name):
    pass

def test_hello():
    assert hello_world("World!") == "Hello World!"

You’ll notice that our test function name begins with the word test. That’s how pytest discovers test methods. Also:

• Test files should be named test_<something>.py or <something>_test.py
• Test classes should be named Test<Something>

Running pytest hello.py should return:

collected 1 item
hello.py F                         [100%]

===================== FAILURES ======================
    def test_hello():
>       assert hello_world("World!") == "Hello World!"
E       AssertionError: assert None == 'Hello World!'
E        +  where None = hello_world('World!')

hello.py:6: AssertionError

Pytest shows why the test failed: AssertionError: assert None == 'Hello World!'. Obviously, we have no code in our hello function so let’s fix that.

def hello_world(name):
    return "Hello {}".format(name)

def test_hello():
    assert hello_world("World!") == "Hello World!"

Running pytest hello.py should now return:

collected 1 item
hello.py .                       [100%]

==================== 1 passed in 0.04 seconds ======================

As you can see, it’s pretty easy to get started with testing in python with pytest. Now that we have everything set up, we’ll be going into more advanced features as the series continues.

Python provides a pretty convenient module called heapq that does that for you. heapq comes with a cool set of inbuilt functions that you can read more about in the docs.

Finding the smallest items

Find 3 of the smallest items using the nsmallest function:

import heapq

numbers = [9, 45, 21, 4, 63, 3, 109]

print(heapq.nsmallest(3, numbers)) # [3, 4, 9]

Also, converting a list into a heap using the heapify function automatically sets the smallest item as the first:

import heapq

numbers = [9, 45, 21, 4, 63, 3, 109]

heapq.heapify(numbers)
print(numbers)
# Output: [3, 4, 9, 45, 63, 21, 109]

You can then pop from the heap with heappop():

heapq.heappop(numbers) # 3
print(numbers)
# Output: [4, 45, 9, 109, 63, 21]

heapq.heappop(numbers) # 4
print(numbers)
# Output: [9, 45, 21, 109, 63]

Finding the largest items

Find 3 of the largest items using the nlargest function:

import heapq

numbers = [9, 45, 21, 4, 63, 3, 109]

print(heapq.nlargest(3, numbers)) # [109, 63, 45]

A more practical example

import heapq

people = [
    {'fname': 'John', 'lname': 'Doe', 'age': 30},
    {'fname': 'Jane', 'lname': 'Doe', 'age': 25},
    {'fname': 'Janie', 'lname': 'Doe', 'age': 10},
    {'fname': 'Jane', 'lname': 'Roe', 'age': 22},
    {'fname': 'Johnny', 'lname': 'Doe', 'age': 12},
    {'fname': 'John', 'lname': 'Roe', 'age': 45}
]

oldest = heapq.nlargest(2, people, key=lambda s: s['age'])
print(oldest)
# Output:
# [{'fname': 'John', 'lname': 'Roe', 'age': 45},
# {'fname': 'John', 'lname': 'Doe', 'age': 30}]

youngest = heapq.nsmallest(2, people, key=lambda s: s['age'])
print(youngest)
# Output:
# [{'fname': 'Janie', 'lname': 'Doe', 'age': 10},
# {'fname': 'Johnny', 'lname': 'Doe', 'age': 12}]

It should be noted though that the nsmallest(n, iterable) and nlargest(n, iterable) functions perform best for smaller values of n. For larger values, it is more efficient to use the sorted() function. Also, when n==1, it is more efficient to use the builtin min() and max() functions.

• Quick sort
• Heap sort
• Counting sort

Again, this is not intended to be an in-depth explanation on the ins and outs of how the algorithms work and their performance. If you’d rather read about that, here’s a nice resource I found: Sorting Algorithms