Linux Tip – Owen's Photolog

Upgrading a Laptop: a few notes

I recently replaced my 2013 Dell XPS 13 with a 2018 Dell XPS 13. This was a very smooth transition, but I thought it was worth publishing some notes on the process. Yes, users of Real Operating Systems are probably laughing, and I wish Ubuntu had a migration tool as nice as Apple’s (assuming theirs still works). But if you’re like me and choose to use a penguin-themed operating system, you might find these notes useful.

This isn’t meant to be a proper HOWTO, more of a list of notes to myself for the next time I do this.

Tips:

Migrating from machine to machine is much easier when they are both running the same OS. In my case, I had Ubuntu 18.04 installed on both machines.
Don’t change your username. Keep it simple!
Screen resolution and hard drive size were the same. Again, minimizing change means your existing preferences will move over perfectly.

Process

Do a basic installation on the new machine, just enough that it’s booting and your user exists.
Make a backup of your home directory. You should use rsync for this. The instructions I followed include steps on how to do this.
Export your package manager configuration, including the package sources and their keys.
Generate a list of all of the packages that were installed on the old machine.
Set up your new machine with the same packages sources and keys
Tell the new machine to install the entire list of packages you saved.
Reboot once
Sync your user data to the new machine
Reboot again
Pray

I didn’t even need to migrate my /etc directory. OK my printer config vanished, but that was easy enough to set up again.

To do all of those steps I followed these instructions. There were some weird packages installed on the old machine that it couldn’t find, but none of them really seemed to make a difference. The new machine was working as of the very first reboot, with the following one major issue, and a few minor issues.

Issues

Major Issue: Battery Drain During Sleep

YIKES. This was not fun to discover. I quickly saw that my new Dell XPS 13 (9370) Laptop running Ubuntu 18.04 was draining battery while sleeping (suspended), at least 5% per hour. It also felt warm even after having been “asleep” for hours. It turns out this is a problem with the default sleep method being set wrong. The default is set to “s2idle” when “deep” is the right answer. See this AskUbuntu post for details about how to check for and fix this problem.

Touchpad

By default, the new touchpad uses two fingers to right click instead of the right side of the pad. I decided to revert back to what I was used to be changing a setting:

gsettings set org.gnome.desktop.peripherals.touchpad click-method areas

Mouse Cursor

The mouse cursor has always been the bane of my existence. I like the DMZ-Black cursor instead of the default white. For some reason there are two places where the mouse cursor theme is set. One is in GNOME settings, but the other is system-wide. So I needed to use gnome-tweaks, and then also:

sudo update-alternatives --config x-cursor-theme

And then reboot.

Printer

Because I hadn’t copied my /etc directory I had to reinstall my printer software.

Conclusion

On the whole the migration was very uneventful. I realize that I have a major advantage, being comfortable with Debian systems and apt commands and rsync, and not everyone can or wants to put up with this kind of procedure. But for an operating system that is still lousy on the desktop, it actually works pretty nicely and is pleasant to use. Here’s to another 5 years before I need to do this again!

Synchronizing large amounts of music to an Android phone

I am a strange person that likes to have local copies of my music. I occasionally find myself driving in rural areas without a cell signal, or on a flight where I don’t want to pay the wifi charge, or on a primitive subway (like in New York) where the tunnels have no reception ((Please refrain from commenting about how you stream all your music over the internet, and get off my lawn)).

Previously in these situations I’d use a portable mp3 player, the late great Sansa Fuze, and that was good enough. But recently my Fuze bit the dust. Since nobody sells mp3 players anymore ((and I can’t use an ipod)) I have to put all my music on my phone. Copying a bunch of files is simple enough, but I sometimes acquire new music and I want a braindead way to update the music on the phone, preferably with rsync. Unfortunately this turned out to be Hard.

First failed attempt: rsync over MTP

I tried to do the most obvious thing first: connect the phone with USB and rsync from the source to the destination. This does not work well at all. Modern phones connect to computers using MTP, or Media Transfer Protocol, which is designed for extremely simple file listing, copying, moving, and deleting. It doesn’t like random reads and it doesn’t handle partial writes. Over MTP, rsync goes intolerably slow as it tries to figure out which files to copy. I tried setting various parameters on rsync to adapt for this, but nothing worked. Fail.

Second failed attempt: FolderSync from Drive to the phone

The second idea was suggested to me by a coworker. I already have all my music on Google Drive, so he suggested I try using the FolderSync app to sync from Drive to my phone. This sort of worked, but after 300 or so files the process would stall, and then halt. I had to cancel the process and start it over again — at least it was smart enough to pick up where it left off. But after doing that five times I gave up. Also, FolderSync and rsync have different ideas about which files were new. FolderSync was recopying files that were already on the destination and I couldn’t configure it to only look at, for example, the file size when determining which files to copy. Fail.

Third mostly failed attempt: rsync and SSHelper

My third idea was to use SSHelper, a neat little Android app that opens up an ssh server on your phone. I would use rsync, like I wanted to, but it would transfer over the network instead of USB. This worked, but transferring the initial 50 gigs of data over the network was going much too slowly. By this point I’d spent two days trying to get my music on my gotdamn phone and was not feeling patient.

Solution: MTP, then rsync and SSHelper

The idea that worked was to use MTP for what it’s meant for: dumping the music folder wholesale from my computer to the phone over USB. Connect phone, drag and drop music onto phone, wait. Now that I’ve seeded my phone with the initial payload, I can use SSHelper and rsync to update the phone with just the new tracks. Listing and comparing files is plenty fast with SSHelper, and copying whatever few files I updated goes quickly enough over wifi.

There you have it! I do miss the days of just loading up a micro SD card and popping it in the phone, but this solution works perfectly well for my needs.

Linux Tip: Reinstalling Ubuntu

Sometimes if a system has been around a long time, upgraded a bunch, perhaps restored from backup, it will start to act up in weird ways. This is the dreaded “cruftiness” that Windows was famous for. You could treat a Windows XP system delicately, and there would still come the day when it was performing so badly it was easier to reinstall the OS than try to fix it.

Well, it can happen to linux, too. I was seeing odd behavior with suspend and resume, and very slow wifi reconnect times, and after dutifully filing a bug I eventually determined the problem was some crufty junk on my system related to a hard drive transfer I’d done.

I thought I was in for a class Linux Weekend — install the new OS, copy my home folder, and then painstakingly reinstall all the programs and packages I use that aren’t in the default installation.

No need! Ubuntu now has a special installation mode called “Reinstall”. It preserves all your user files while cleaning out the system folders completely. Then it installs a fresh OS and tries to install as many of your programs as it can find.

After I rebooted, there were a tiny number of custom-installed programs that the system couldn’t reinstall, but otherwise everything was where I left it and suspend and resume now work perfectly. Ah yes, and all of my wifi access points have been forgotten. For something I just let run over night, this was relatively painless and worked great! 5 stars.

Streaming audio over the network in linux

I’ve been trying for a long time to find a simple (for me) way to get sound from my laptop to my TV. I’ve tried mpd, mediatomb, pulseaudio… basically everything short of spending actual money. Finally I found a low-overhead system that does what I need thanks to our friends ssh and alsa: Linux Network Sound on the Very Cheap. Thanks so much to Aristotle Pagaltzis for this recipe.

I’ll reproduce the instructions with my edits:

On the source machine (my laptop in this case), load the loopback ALSA driver:
```
modprobe snd-aloop index=1 pcm_substreams=1
```
The driver provides a card with two sound devices, and when sound is output onto a stream on one device then the driver loops that back that as an input available on the same stream on the other device. You’ll need to pick an index that’s not already taken by a sound card on your machine or you’ll get an error.
Add this to your ~/.asoundrc:
```
pcm.loop {
  type plug
  slave.pcm "hw:Loopback,1,0"
}
```
The original instructions also tell you to change your default soundcard to the loopback device. In this case I don’t want all my sound to go over the wire. Instead, use pulseaudio to select an application and have its sound sent to the loopback device. Yes, I know, pulseaudio. Actually it works pretty well these days! The loopback driver will then make that audio available on its other stream.
On the machine with speakers you can then do this:
```
ssh -C user@hostname sox -q -t alsa loop -t wav -b 16 -r 48k - | play -q -
```
This command causes a simple dump to be made of the “loop” device on the source machine. That audio data is then piped into the “play” command which causes the sound to play on the machine connected to my TV. This also sets up ssh with compression which helps keep the bandwidth usage down (about 200K/s for me).
All you need to do then is play something from the laptop, open the pulseaudio volume control, and tell the output to go to the loopback device. Very quickly the audio pops out the other side.

Obviously this is no solution for people wanting multiple client control, tablet support, playlists, or anything fancy like that. But as a way of just getting audio from point A to B, this is all I need.

Git Cheatsheet

Git has a steep learning curve but persistence pays off. Now that I’ve got the hang of it I can slosh code changes between multiple branches like a motherfucking wizard. Here is a cheatsheet I’ve developed for myself that has helped a lot. This isn’t a complete git HOWTO, it’s more of a list of things that I kept doing wrong as I was learning the system. The tips start relatively basic and get more and more complex. I was coming from svn and bzr so a lot of my tips have to do with confusion that results in coming from those systems.

Rule #1: When in doubt, checkout!

set username:

git config --global user.email user@domain git config --global user.name 'Owen Williams'

set merge tool:

git config --global merge.tool meld

checking out:

git clone [url]

update changes from remote branch:

git pull (does git fetch and then merges in the changes)

revert a file:

Don’t use “reset”!
git checkout [file]

show a single diff:

git show [sha]

create a branch:

(Don’t actually use this, because the next command is better)
git branch [newbranch]

create a branch and switch to it:

git checkout -b [newbranch]
(-b runs git branch first to create the branch. You could also git branch [newbranch] and then git checkout [newbranch].)
This works even if you’ve started working! The changes will be moved over to the new branch 🙂

list branches:

git branch -a

switching branches:

git checkout [branch name]

merge all changes from another branch:

git merge [branch]

merge single commit from another branch:

(Only use this as a last resort, you lose wondering commit-tracking this way)
git cherry-pick [sha]

merge individual files from another branch:

git checkout source_branch [files]

fixing conflicts:

git mergetool
This uses the configured diff tool, or -t [toolname].
Note that with meld, the file called “BASE” (in the center) is the one that is used for the final commit. This is handy because you can pick which edits from the local you want, and which ones from the remote you want. Or just edit the <<< >>> file like you usually do. I find a good way to work is to blindly merge everything from the REMOTE to BASE first. Then I can see what’s different between the REMOTE and LOCAL which is usually what I want to do.

undo a merge in progress:

git merge --abort

reverting a commit:

You’re in trouble now, but this can be done. Examine git reflog which lists all commits for all branches. You want to rewind to before the bad commit.
git reset HEAD@{2} (or whatever label)
Default reset is –mixed, which seems to be ok. don’t do a –hard, like, ever.
When you do a reset, you’ll still be in a weird place with a lot of weird commits, but at least you saved your log. Examine the files and do a commit right away.
DO NOT DO git reset --soft [SHA] from trunk if you’re in a branch! — this will BLOW AWAY your original log of changes, ie how you got to this point in your branch. You’ll have all those changes, but none of the intermediate work.

create a new branch from current that tracks a remote branch other than master:

need to create a local branch that tracks the remote branch
don’t include “remotes/”
git checkout --track -b mybranch origin/branch

tell git that the current branch should always inherit changes from another branch

git branch -u [parent branch]
git pull is familiar for pulling from a remote repo, but it also works between branches. This is great for hierarchical featuresets, ie:

feature1
feature1-feature2

So let’s say feature1 is in code review, I can keep that clean but keep working on the next feature. I can tell feature1 to pull from master, and feature1-feature2 to pull from feature1. (I name my branches like that so I don’t get lost in the tree.) After doing this to every branch, I can run git pull no matter where I am and everything will be in sync.

copy one repository to another:

Ok so this is trickier, but possible.

first, clone the new repo.
Add the other repo as a remote. here “other-repo” is the name of the new remote branch.
git remote add other-repo [other branch location] git fetch other-repo git checkout other-repo/master
Since you can’t actually check out a remote branch, you’re in limbo. Give it a name, creating a place where the other master branch will reside. This is where you’re doing the merge from the old into the new repo.
git checkout -b other_master git merge master
resolve any conflicts between the two repos. Trust the LOCAL changes, not remote! you want the new master to be in charge, yes?

Now for every branch on the other repo you care about:
check it out
git checkout other-repo/branchname
copy that to local
git checkout -b branchname
merge in the changes that bring it in line with current master:
git merge other_master

then to check your work, compare against new master
git diff master

also notice that all your commits are preserved!
git log

HOWTO: Change the terminal title when commands are running

In the work I do I often have a lot of terminals open, and some of those terminals have a lot of tabs. Sometimes I start a process that’s going to take a long time and I’d like to quickly see if that process is still running or if it has finished. While this is tricky, it’s possible! This HOWTO is bash-specific and is designed for Ubuntu, but it should work on other distributions with minimal changes. Thanks to this url for getting me started: http://superuser.com/questions/42362/gnome-terminal-process-name-in-tab-title.

Step one is to make sure that your current settings do not change the title of the terminal. On Ubuntu there’s a block of code in the ~/.bashrc file that says “If this is an xterm set the title to user@host:dir“. Just comment out that whole block by putting # characters at the beginning of each line to disable the functionality.

# If this is an xterm set the title to user@host:dir
#case "$TERM" in
#xterm*|rxvt*)
#    PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
#    ;;
#*)
#    ;;
#esac

On other systems you will have to look around for where the value of PS1 is set. Look for “\e” — this is the code that tells bash what title you want to use. We want to override this, so delete everything from \e to \a.

Step two is to insert new code into ~/.bashrc (or wherever your system prefers). I put this code right after the existing PS1 code that Ubuntu shipped.

# Custom title-setting code that adds a triangle play-arrow
# if the terminal is not waiting on the prompt
case "$TERM" in
xterm*|rxvt*)
    # This tells bash: before showing the prompt, run this
    PROMPT_COMMAND='echo -ne "\033]0;${THIS_TERMINAL_TITLE}\007"'

    # Edit the title if a command is running:
    # http://www.davidpashley.com/articles/xterm-titles-with-bash.html
    show_command_in_title_bar()
    {
        case "$BASH_COMMAND" in
            *\033]0*)
                # The command is trying to set the title bar as well;
                # this is most likely the execution of $PROMPT_COMMAND.
                # In any case nested escapes confuse the terminal, so don't
                # output them.
                ;;
            *)
                echo -ne "\033]0;▶ ${THIS_TERMINAL_TITLE}\007"
                ;;
        esac
    }
    # The DEBUG signal simply announces the last-run command
    trap show_command_in_title_bar DEBUG
    ;;
*)
    ;;
esac

# Now we can set the title of the terminal with terminal_title my title
function terminal_title ()
{
    export THIS_TERMINAL_TITLE="$@"
}
# Here's a good default
export THIS_TERMINAL_TITLE="$USER@$HOSTNAME"

This code works because of a few obscure features in the bash shell. The first is the existence of $BASH_COMMAND — a variable that holds the value of the currently-running command. Second, the trap command which allows us to take action when commands are executed, and third, the $PROMPT_COMMAND variable which tells bash to execute a command before showing the command prompt. With these features, we can change the title when commands are executed and change the title back when the commands are done executing.

In my case all I do is take the current terminal title and add a play-like triangle character to the beginning while commands are running. When the commands are complete the old title is restored and the triangle disappears. One could extend this feature by taking the value of $BASH_COMMAND and adding it to the title. By default the terminal title is just $USER@$HOSTNAME but I’ve added a function which allows me to change the title to anything I want.

Step three requires you to either log out and log back in for the changes to take effect, or to run source ~/.bashrc in order to execute the commands. Note that some terminals have to be set up properly to allow their titles to be changed. In gnome-terminal, go to Edit / Profile Preferences / Title and Command and make sure “When terminal commands set their own titles” is set to anything except “Keep initial title.” Also, if you have set a custom title in gnome-terminal itself by right clicking a tab and selecting “Set Title…”, you’ll need to erase that custom title by selecting “Set Title…” and erasing the text inside there. This can be a little tricky, and logging out may be the simplest solution. If you’re still not seeing the terminal title change, check echo $PS1 and make sure there is no \e visible.

Now when I run a long compile or test command, I see a ▶ in the titlebar and tab of my terminal. When the triangle disappears I know my command has completed. Please share your favorite extensions to this feature in the comments!

Linux Tip: Stuttering Audio with Flash and Firefox

Problem: When playing back flash video in firefox on Ubuntu version 12.04, you may experience stuttering audio and dropped frames
Possible cause: pulseaudio may be configured for “simultaneous output”, which adds overhead that may be slowing down your system.

Solution (in my case): Click the volume icon, then click Audio Settings. Make sure the default output device is the correct sound card, not “Simultaneous output.” Also make sure the settings for the sound card are correct, including inputs and codecs. Don’t turn on anything you don’t need.

Keybinding GTK to shifted numbers

GTK (and I think QT also) has a fairly intuitive system for binding keys to commands: Control 3 is just "<Ctrl>3". However when you want to bind a key with Shift, things get tricky. I had tried "<Ctrl><Shift>3", and "<Ctrl>#", and "<Ctrl><Shift>#", all without success.

To make this particular keybinding work, you have to use the special X11 keysym name for that key. I found permanent url that describes these keys here, and a better list with printable characters here.

So for my example, the proper binding would be "<Ctrl><Shift>numbersign".

However keybindings like this are very tricky and shouldn’t be used in production applications (this example is a personal plugin for GEdit). This is because keyboards in other countries have different symbols over different numbers. So it’s not a given that Shift-3 is the same as numbersign in every country.

Ubuntu sometimes launches the wrong web browser

I just upgraded to Ubuntu Oneiric 11.10, and aside from being forced to abandon Gnome 2 for Unity or Gnome 3, I found an annoying bug that took me a while to fix. Although I have set up Firefox as my default web browser, I was finding that sometimes Epiphany would get launched instead. Specifically this problem turned up in my app PenguinTV.

The reason for this is that although there is a per-user “preferred application” setting, there is also a system-wide setting that can only be changed on the command line. Run this command in a terminal:

sudo update-alternatives --config gnome-www-browser

This will give you a list of browsers to choose from, one of which will be the one you actually want.

Reordering Indicators in Ubuntu

Now that the GNOME panel is dead (long live the panel), the next best option for displaying system information in the upper menu bar is Ubuntu’s Indicators. Indicators are a much less powerful system than the old panel once was, due to various tradeoffs of complexity for ease-of-use. One limitation is that indicators seem to appear in a random, arbitrary order. I’d prefer to arrange the indicators myself. For instance I’d like to have all my system monitors grouped together.

It turns out there is a way to reorder the icons, but it’s tricky. These instructions assume you know how the command line works and can read a process list without getting worried. Until the Ubuntu developers add the ability to reorder indicators this is the only way to do it. This page gave me a good head-start. You have to start by creating a special “override” file that will tell the indicator applet how to reorder the icons. Start by creating the override file with the system defaults:

mkdir -p ~/.local/share/indicators/application cp /usr/share/indicator-application/ordering-override.keyfile ~/.local/share/indicators/application/ gedit ~/.local/share/indicators/application/ordering-override.keyfile &

My default file looked like this:
[Ordering Index Overrides] nm-applet=1 gnome-power-manager=2 ibus=3 gst-keyboard-xkb=4 gsd-keyboard-xkb=5

The next step is tricky: You’ll need to add each indicator to the file and assign it a number (lower numbers are further to the right), but what do youo call each item? In many cases (but not all), the name is the same as the executable. indicator-cpufreq is just indicator-cpufreq. Run this command to list all the indicators running on your system:

ps xa |grep indicator-

Other applications, like Tomboy, set up their own indicator name. We need a way of finding out what that name is.

The only way I know of to find this name is to rerun the indicator service and read through the debugging output. Yeah, I know. Warning, this process may mess up the display of your indicators. Nothing done here is permanent, so you can always log out and log back in to restore your regular desktop.

Try this:
killall indicator-application-service && /usr/lib/indicator-application/indicator-application-service

Much text will start spewing. If the command prompt comes back right away and you see the following text, just try the command I gave you again:

(process:7601): libindicator-WARNING **: Name request failed. (process:7601): indicator-application-service-DEBUG: Service disconnected (process:7601): indicator-application-service-DEBUG: Application proxy destroyed '(null)' (process:7601): indicator-application-service-DEBUG: Application free '(null)'

Look for lines like this:
(process:7672): indicator-application-service-DEBUG: 'bluetooth-manager' ordering index is '20626C75'

This tells you two things: bluetooth-manager is the name of this particular indicator, and that’s what you’ll want to put in your override file. Also, 20626C75 is the current ordering index. Because it’s a long number, this value is not currently being overridden.

Now we want to make some changes.

Edit your override file. Here’s mine:
[Ordering Index Overrides] nm-applet=1 gnome-power-manager=2 ibus=3 gst-keyboard-xkb=4 gsd-keyboard-xkb=5 indicator-penguintv=6 tomboy-notes=7 multiload=8 indicator-sensors=9 indicator-cpufreq=10 indicator-sysmonitor=11

The rerun the command I gave you. You should see some changes:

(process:7672): indicator-application-service-DEBUG: 'indicator-sensors' ordering index is '9'

This means that I successfully assigned the ordering value of 9 to indicator-sensors. And indeed, all my indicators are in the order I want.

When everything is how you want it, log out and log back in to make sure the changes worked.

update 10/19/2012:

there’s a better way to list running indicators:
dbus-send --type=method_call --print-reply \ --dest=com.canonical.indicator.application \ /com/canonical/indicator/application/service \ com.canonical.indicator.application.service.GetApplications | grep "object path"

This will print out the running indicators. replace underscores with hyphens in the keyfile