Here's the command to upgrade it. Took me too long to figure it out, so I thought I'd share.

postgres@hostname:~$ /usr/lib/postgresql/9.5/bin/pg_upgrade -b /usr/lib/postgresql/9.4/bin -B /usr/lib/postgresql/9.5/bin -d /var/lib/postgresql/9.4/main/ -D /var/lib/postgresql/9.5/main/ -o "-c config_file=/etc/postgresql/9.4/main/postgresql.conf" -O "-c config_file=/etc/postgresql/9.5/main/postgresql.conf"