Convenient method to connect to Jupyter notebooks on a private server over SSH

This function allows you to run a single command from your laptop that will start the Jupyter notebook server on a remote machine (unless it's already started), and then set up the necessary port forwarding over SSH. Once you run this command, you can navigate to localhost:8889 on your laptop and you can work with the notebooks on the remote server. The server will need the screen utility to be installed.

function rj() {
  # connects to remote jupyter notebook server 
  ssh USERNAME@SERVER_ADDRESS 'screen -dmS jupyter-notebook bash -c "path/to/jupyter notebook --no-browser --port=8892 --port-retries=0"'

  # kill any stale connections
  existing_process=`ps aux | grep -v grep | grep "fNL localhost" | awk '{print $2}'`
  if [[ ! -z "$existing_process" ]]; then 
    kill $existing_process 

  # connect to server and forward some ports
  ssh -fNL localhost:8889:localhost:8892 USERNAME@SERVER_ADDRESS

Tradeoffs and caveats: anyone else with SSH access to the server will be able to see your notebooks. Each user will also need their own, unique port.

A temporary fix to several audio bugs in Ubuntu

There are two bugs affecting my computer that prevent audio from playing:

  1. If I boot my Ubuntu laptop with headphones plugged in, no audio will play while the headphones are plugged in.
  2. Firefox 58 and 59 will not play audio unless another program produced a sound before Firefox was started.

I finally found a combination of hacks that resolves these issues, though this is clearly only for the kind of sociopath that continues to use Linux under such conditions.

Install mpg123: sudo apt install mpg123

Download this 50 ms inaudible mp3 I made: curl -o ~/Documents/very-quiet-noise.mp3

Then add this to ~/.profile:

# Plays a sound at boot so Firefox 58+ will be able to play audio.
mpg123 -q  ~/Documents/very-quiet-noise.mp3

# If we booted with headphones plugged in, this will allow audio to work
# The strange invocation is required to suppress an error message on boot
bash -c 'alsactl restore || true' 2>&1 > /dev/null

Custom scatterplot colors with colorbar in Matplotlib

I needed to plot a standard scatterplot, but color the points according to some third dimension of data, and add a colorbar. This was surprisingly not documented anywhere I could find.

import matplotlib
import matplotlib.pyplot as plt
import random

# We have three dimensions of data. x and y will be plotted on the x and y axis, while z will 
# be represented with color.
# If z is a numpy array, matplotlib refuses to plot this.
x = list(range(300))
y = sorted([random.random()*10 for i in range(300)])
z = list(reversed([i**3 for i in y]))

# cmap will generate a tuple of RGBA values for a given number in the range 0.0 to 1.0 
# (also 0 to 255 - not used in this example).
# To map our z values cleanly to this range, we create a Normalize object.
cmap ='viridis')
normalize = matplotlib.colors.Normalize(vmin=min(z), vmax=max(z))
colors = [cmap(normalize(value)) for value in z]

fig, ax = plt.subplots(figsize=(10,10))
ax.scatter(x, y, color=colors)

# Optionally add a colorbar
cax, _ = matplotlib.colorbar.make_axes(ax)
cbar = matplotlib.colorbar.ColorbarBase(cax, cmap=cmap, norm=normalize)


Executing long-running Jupyter notebooks in the background

I have some notebooks that take over 12 hours to complete and use enough memory that they must be run on a large server. While you could set up VNC server or something to have a persistent, interruptible session, I prefer this short function that I just run in a screen session. It's pretty straightforward, except for --ExecutePreprocessor.timeout=-1 which is necessary since each cell has only 30 seconds to finish running by default.

function runbook() {
    jupyter nbconvert --to notebook --ExecutePreprocessor.timeout=-1 --execute --inplace $1

Then it's as simple as running:

runbook my-long-running-notebook.ipynb

Eliminating freezing in VMWare on Xubuntu

When running Xubuntu in VMWare on an Xubuntu host, the whole virtual OS would freeze for up to minutes at a time, especially after upgrading the host to Xubuntu 17.10. I've updated my GRUB settings as follows and that seems to have resolved the problem:

In /etc/default/grub I changed this line:



GRUB_CMDLINE_LINUX_DEFAULT="quiet splash transparent_hugepage=madvise"

Using compound data types in h5py

Compound data types allow you to create Numpy arrays of heterogeneous data types and store them in HDF5. For the example in this blog post, we want to store X and Y coordinates (as unsigned 32-bit integers), an intensity value (as a float), and a DNA sequence (as a variable-length string).

import numpy as np
import h5py

my_datatype = np.dtype([('x', np.uint32),
                        ('y', np.uint32),
                        ('intensity', np.float),
                        ('sequence', h5py.special_dtype(vlen=str))])
h5 = h5py.File("my_data.h5", "w")
data = [(1003, 4321, 43.2, "ACGTACTG"), (55, 4098, 12.1, "GGT"), (3209, 909, 59.7, "ACTAC")]
dataset = h5.create_dataset('/coordinates', (len(data),), dtype=my_datatype)
data_array = np.array(data, dtype=my_datatype)

We can look at data_array and see how this datatype looks:

In [1]: data_array
array([(1003, 4321,  43.2, 'ACGTACTG'), (  55, 4098,  12.1, 'GGT'),
       (3209,  909,  59.7, 'ACTAC')],
      dtype=[('x', '<u4'), ('y', '<u4'), ('intensity', '<f8'), ('sequence', 'O')])

To save the data efficiently, just assign it to the dataset like so:

dataset[...] = data_array

Regular indexing works:

In [2]: h5['/coordinates'][0]
Out[2]: (1003, 4321,  43.2, 'ACGTACTG')

Selection syntax works basically as you'd expect. Optionally, you can tack on an extra index with fields in it to limit the fields that are returned:

selector = (h5['coordinates']['x'] > 1000) & (h5['coordinates']['intensity'] > 40.0)
fields = ['x', 'y']
coords = h5['/coordinates'][selector][fields]

coords is now a compound datatype consisting of just two 32-bit unsigned integers:

In [3]: coords
array([(1003, 4321), (3209,  909)],
      dtype=[('x', '<u4'), ('y', '<u4')])

Even though coords is homogenous, many numpy operations can't be performed on it. This might not be the best way to do this, but you can get a regular array with:

In [4]: raw_coordinates = np.stack((coords['x'], coords['y']), axis=-1)
In [5]: raw_coordinates
array([[1003, 4321],
       [3209,  909]], dtype=uint32)

Fixing the color of projectors on Ubuntu

When I used to connect my laptop (running Xubuntu 17.04) to a projector, the color would always be slightly yellow and dim. This did not happen with regular external monitors.

My solution was to use xcalib. To install:

apt install xcalib

Then, while connnected to the projector, run:

xcalib -c

While it's just a hypothesis, I believe what happened was that I was running some program to eliminate blue colors at night, and I still had that running when I first connected to a projector. Then my computer may have permanently associated that color scheme with all external projectors.

DS9 Viewing Guide

I loved Star Trek: Deep Space Nine, but it really takes a long time for it to come into its own, and there are a lot of terrible episodes, especially in the first few seasons. However, since DS9 has a long-term plot and since there are a lot of great episodes in every season, I wrote this guide so friends could know which episodes they could skip and which ones they really do need to see.

Bold and underlined episodes are critical to the long-term plot or character development
Italic episodes are good and worth watching but do not affect the long-term plot
Red episodes are terrible
All other (unlisted) episodes are neither good nor bad, watch them if you're just really into the show.

World building episodes from Star Trek: The Next Generation

These episodes set the stage for DS9.

S4E12 - The WoundedIntroduces the Cardassians, one of the main antagonists in DS9. Also features O’Brien and Worf, who become main characters in DS9. The actor that plays the Cardassian captain in this episode plays a different Cardassian in DS9, but character-wise they’re very similar.
S5E03 - Ensign Ro: Introduces the Bajorans, who basically had a multi-decade occupation/holocaust done to them by the Cardassians. Ensign Ro was supposed to appear on DS9 but the actress declined; the character Kira is essentially the same person.
S6E10/11 - Chain of Command: Allegory to 1984. One of the highest rated episodes of TNG.
S7E24 - Preemptive Strike: More good background, terrorism stuff.

Season 1

01/02 - Emissary: First episode so mandatory. Some context: the reason Sisko doesn’t like Picard is because in an episode of TNG, Picard was kidnapped by cyborgs (the Borg), turned into a cyborg, and he became their leader and went on a killing spree (which killed Sisko’s wife) before ultimately being saved and reverted to a human.
03 - Past Prologue: Introduces Garak, my favorite character.
08 - Dax: Character building for Jadzia, and they explain the whole Trill thing in detail. Not a great episode but the multi-life thing comes up a lot.
10 - Move Along Home: So awful, do not watch this even for ironic reasons
11 - The Nagus: The first Ferengi-heavy episode. Typically, these serve as comic relief when the show gets really dark, and they’re generally good episodes. Still I’m going to mark them as optional. But this one is a good one to watch just because so many of the main characters are Ferengi, and a lot of them defy their culture, so it’s good to know where they come from.
13 - Battle Lines: Nice episode about war and peace, important development with the Kai.
14 - The Storyteller: Someone on the Internet wrote an essay about why this is the worst episode of any Star Trek series. It was long. Needless to say, skip this one.
15 - Progress: Eminent domain, former rebel realizes she is now the establishment. Kira character-building.
19 - Duet: Great episode about war criminals.
20 - In the Hands of the Prophets: Allegory for teaching evolution in schools, imposing cultural values onto others, terrorism, power struggles. Two important characters are introduced.

Season 2

01-03 - The Homecoming/The Circle/The Siege: An insurgent group engages in shenanigans.
05 - Cardassians: Good Garak episode
06 - Melora
07 - Rules of Acquisition: The Ferengi go to the Gamma quadrant to do stuff. Some important foreshadowing but no game-changers.
13 - Armageddon Game: One-off about biological weapons and the power/danger of information.
14 - Whispers: A nice one-off episode, pretty science-fictiony.
16 - Shadowplay: Another nice one-off episode, but includes some more foreshadowing.
18 - Profit and Loss: Cardassian rebels, Garak and Quark.
19 - Blood Oath: There’s a really good line in this episode that I think of often
20/21 - The Maquis: Rebels, war, secret plans, etc.
(highly recommended) 22 - The Wire: Mostly about addiction, but also goes deep into Garak’s spy-related past.
24 - The Collaborator: Power struggle to become the next space pope, new information on the culprit behind a massacre that occurred during the occupation of Bajor
25 - The Tribunal: O’Brien is accused by the Cardassians of a crime he didn’t commit. Nothing is as it seems.
26 - The Jem’Hadar: One of the defining episodes of the show. The war (effectively) begins.

Season 3

01/2 - The Search: Major plot point, great action
03 - The House of Quark: Quark’s ex-wife rolls into town, shenanigans ensue
05 - Second Skin: Kira might be a secret Cardassian
06 - The Abandoned: They find a baby Jem’Hadar
(good) 07 - Civil Defense: They accidentally trigger a program intended to thwart slave uprisings from back when the station was a mining facility run by the Cardassians
09 - Defiant: Major plot point.
11/12 - Past Tense: I’m not actually that into these episodes, but here is a quote from Wikipedia about it: “as this episode was finishing production an article appeared in the Los Angeles Times describing a proposal by the mayor to create fenced-in "havens" for the city's homeless, to make downtown Los Angeles more desirable for business. The cast and crew were shocked that this was essentially the same scenario that Past Tense warned might happen in three decades, but was now being seriously proposed in the present.”
13 - Life Support: Major plot point, nature of conscious experience
14 - Heart of Stone: I don’t like the main plot of this episode but it’s relatively important, though the B-story is great.
15 - Destiny: Literal interpretations of religious texts can be problematic
16 - Prophet Motive: Decent Ferengi episode, a total one-off
17 - Visionary: Just a good episode about time travel

A few more episodes from Star Trek: The Next Generation

To better know the Klingons and Romulans, who will be playing a much larger role in the show soon, I highly recommend watching a few episodes from The Next Generation:

S2E08 - A Matter of Honor: (the most quotable episode of TNG)
S3E10 - The Defector
S4E26/S5E1 - Redemption: To set this up, Worf’s father was falsely accused of helping the Romulans commit a massacre decades ago, which the Duras family actually orchestrated. If this were to become public knowledge however, it would lead to a civil war, so after learning the truth, Worf agreed to publicly accept that his father was to blame in order to save the Empire. He later stabbed the head of the Duras family to death, but he was the only person who could have restored Worf’s honor. So Worf is stuck in this weird limbo where all Klingons hate him and there’s nothing anyone can do.

Season 3 (continued)

20 - Improbable Cause: Major plot point, great ep
21 - The Die is Cast: Major plot point, great ep
23 - Family Business: Women’s rights
24 - Shakaar: Space pope power struggles and rebels
26 - The Adversary: War episode

Season 4

01/02 - The Way of the Warrior: New characters, major plot points
04 - Hippocratic Oath: Philosophy of giving medical care to the enemy during a war
05 - Indiscretion: They search for a missing ship
07 - Starship Down: Good ep
(highly recommended) 08 - Little Green Men: They go back in time, fun one-off episode
10 - Our Man Bashir: Holodeck shenanigans
11/12 - Homefront/Paradise Lost: The fragility of democracy during a time of war and suspicion
13 - Crossfire: Seriously I hate this entire subplot but it matters to the show
14 - Return to Grace: A little slow but important developments occur
15 - Sons of Mogh: Morality of assisted suicide
16 - Bar Association: Fun Ferengi episode about labor rights and unions
17 - Accession: Faith, caste systems, etc. I don’t care for this kinda stuff
(recommended) 18 - Rules of Engagement : Worf commits a war crime but you aren’t supposed to do those
(recommended) 19 - Hard Time: Very science-fictiony story about a novel way of punishing prisoners
22 - For the Cause: Sisko’s girlfriend might be fooling around on him
23 - To the Death: Renegade Jem’Hadar
(good) 24 - The Quickening : Bashir helps with a plague, allegory for the third world in some respects
(good) 25 - Body Parts: Ferengi episode
26 - Broken Link: Shit gets real

Season 5

01 - Apocalypse Rising
02 - The Ship
03 - Looking for par’Mach in All the Wrong Places
(recommended) 04 - Nor the Battle to the Strong: MASH in space
(recommended) 05 - The Assignment: O’Brien can’t catch a break
(recommended) 06 - Trials and Tribble-ations: Only recommended if you have seen the original Tribbles episode from original Star Trek
(recommended) 09 - The Ascent: The Odd Couple in desolate wilderness
10 - The Rapture: Sisko has prophetic visions
11 - The Darkness and the Light: Someone is murdering Kira’s friends
12 - The Begotten: Major development
(recommended) 13 - For the Uniform: A little ham-fisted with the allegory, but Sisko goes full beast mode
14 - In Purgatory’s Shadow: One of my favorite episodes of the entire series, very important, do not read the Netflix synopsis, don’t look at the preview screen cap
15 - By Inferno’s Light: Major plot point, do not read the Netflix synopsis, don’t look at the preview screen cap
16 - Doctor Bashir, I Presume?: Not that important but a secret about Bashir is revealed that comes up repeatedly from here on out
(recommended) 18 - Business as Usual: Quark gets involved with an arms dealer
19 - Ties of Blood and Water: More war crimes deliberations, callback to S3E05 “Second Skin”
(good) 20 - Ferengi Love Songs: Good Ferengi ep, as all Ferengi eps are
(good) 21 - Soldiers of the Empire: Good Klingon episode
(good) 22 - Children of Time: Science fictiony episode
(good) 23 - Blaze of Glory: Stolen nukes
(good) 24 - Empok Nor: Spooky episode
25 - In the Cards: Important strategic things with a nice comedic relief subplot
26 - Call to Arms: Shit gets real, but for real this time

Season 6

01 - A Time to Stand
02 - Rocks and Shoals: This is the most metal of all episodes. Two of my hypothetical cats are named after characters in this episode, an honor not frequently bestowed. Try to guess who!
03 - Sons and Daughters
04 - Behind the Lines
05 - Favor the Bold
06 - Sacrifice of Angels
07 - You Are Cordially Invited…: If you watch this episode and get married and still don’t have a Klingon-themed wedding then you are history’s greatest monster
09 - Statistical Probabilities: Flowers for Algernon in space
10 - The Magnificent Ferengi: Yes, a non-optional Ferengi episode! Very war-related
11 - Waltz
(recommended) 12 - Who Mourns for Morn?
13 - Far Beyond the Stars: One of the highest-rated episodes by DS9 fans. I don’t think it’s the greatest but I do like it, and it’s pretty unusual. And you get to see the actors without prosthetics. There’s also a brief reference to it in season 7 that will be super confusing if you haven’t seen it
(recommended) 14 - One Little Ship: Just a fun one-off episode that makes a really good point about a fatal flaw in a well-known children’s movie
(recommended) 15 - Honor Among Thieves
(recommended) 16 - Change of Heart
(recommended) 17 - Wrongs Darker than Death or Night
18 - Inquisition: Major downstream plot points based on this episode
19 - In the Pale Moonlight: Major plot point, Garak goes full Garak. One of my favorite episodes. Perhaps my favorite of all.
20 - His Way: I just don’t like anything about this subplot, but it does have large implications for several characters. And they introduce a holodeck program that will be featured repeatedly. Protip: you can skip the music in every episode, nothing important ever happens during a song
21 - The Reckoning
(recommended) 22 - Valiant: Hubris, ambition, elitism
(recommended) 23 - Profit and Lace: Women’s rights
(not that great) 24 - Time’s Orphan: Another terrible thing happens to O’Brien
(okay-ish) 25 - The Sound of Her Voice: Mild science fiction
26 - Tears of the Prophets: Important war episode

Season 7

01 - Image in the Sand
02 - Shadows and Symbols
03 - Afterimage
04 - Take Me Out to the Holosuite: So poorly written and executed, everyone involved with this episode should be bricked up inside a wall
05 - Chrysalis
06 - Treachery, Faith and the Great River
07 - Once More Unto the Breach
08 - The Siege of AR-558
09 - Covenant
10 - It’s Only a Paper Moon
11 - Prodigal Daughter: A few bad things happen to O’Brien, and he’s not even the focus of the story. Held by the writers to be the worst episode of the season, presumably because no one wants to mention episode 704 out loud for fear of what happens to those who do. I thought it was fine.
14 - Chimera: Odo meets a douchebag Changeling
(good) 15 - Badda-Bing, Badda-Bang: Oceans Eleven in space
16-26: The last 10 episodes are effectively one continuous plotline, and they’re all great.

Coherent Diagnostics

I had to get some diagnostic information from a Coherent laser. It's super easy if you connect via USB on an Ubuntu machine.

First, install minicom:

sudo apt install minicom

Now you need to find the name of the device. Run

dmesg | grep tty

The output will look something like this:

$ dmesg | grep tty
[    0.000000] console [tty0] enabled
[26890.843225] usb 1-2: FTDI USB Serial Device converter now attached to ttyUSB0

Unplug the USB and plug it back in again, and rerun

dmesg | grep tty

You'll see additional messages about something getting disconnected and attached, similar to this:

$ dmesg | grep tty
[    0.000000] console [tty0] enabled
[26890.843225] usb 1-2: FTDI USB Serial Device converter now attached to ttyUSB0
[27005.835517] ftdi_sio ttyUSB0: FTDI USB Serial Device converter now disconnected from ttyUSB0
[27057.953214] usb 1-2: FTDI USB Serial Device converter now attached to ttyUSB0

In this example, the device name is ttyUSB0. Now run

sudo minicom -s

and select “Serial port setup”. Type A (no enter key, it will automatically move your cursor) and change the serial device to


or whatever your device name is. It will have /dev/ in front of it no matter what.

Now type E to set up the other parameters. Type B to lower the speed (or A to raise it) to 19200. It says the current speed at the top. Type Q to set the parity and stop bits. When you're done, it should say at the top:

Current: 19200 8N1

If it does, press enter. Now choose Exit (not Exit from Minicom). You'll be taken to a terminal where you're now connected. You can start issuing commands, which all start with a question mark. Try:


It should return some sort of number or something. The rest of the commands should be saved here, or you can ask Coherent for them.

Mostly colorblind-friendly palette