thinkweekly

bash history forever

2008-05-17T21:48:00.000-07:00

In what seems eerily like a burst of structured procrastination I recently got around to a task I'd been meaning to do for ages: infinite, global bash history across all of my shells.
I started by reading over how the history mechanism in bash is supposed to work, and then a co-worker and I set about writing the simplest hack that could actually record all my command lines and aggregate them for me.
Step 1 is to make sure that every typed command is recorded somehow. Simply extending my PROMPT_COMMAND to be a function that would call other functions is a good starting place:


export PROMPT_COMMAND='process_prompt'

function process_prompt {
  write_last_command
  write_global_history
  read_global_history
  xsettitle $HN $(shortenpath $PWD)
}

I'm going to ignore the titlebar setting stuff as it's not related to this post. As you can see, the basic structure at the top level is to record the last command typed in the current shell; recompute the global history; and then load the global history into this shell. Inefficient? Horribly so. Can I tell? Not on today's monster workstations. Now, to record the commands of the current shell requires a file to write to:


COMMANDFILE=~/.histories/$HN.$$
trap "rm -f $COMMANDFILE" INT TERM EXIT

where in the above snippet $HN is my hostname after processing. If you're actually following along you will also want a some lines like this:


export HN=$(hostname)
HN=${HN%%.*}

to set the hostname. The commands from each file, then, are to be written to my home directory in a file named .histories/hostname.processid that will get deleted when the current shell exits. The actual writing of each command line is, amusingly enough, accomplished via the history bash built-in:


function write_last_command {
  echo $(date +%s).$(history 1 | sed -e 's/^[ 0-9]*//') >> $COMMANDFILE
}

Then, the next step is to merge all the files in ~/.histories into a master file:


function dedup_history {
  sort -nr | sort -t\. -k 2 -s -u | sort -n
}
function write_global_history {
  cat $(find ~/.histories -type f) | dedup_history |
     sort -t\. -k 2 -s -u | sort -n > ~/.histories/global.$$
  mv ~/.histories/global.$$ ~/.histories/global
}

And finally, load the history into RAM:


function read_global_history {
  sed -e 's/[0-9]*\.//' < ~/.histories/global > $HISTFILE
  echo '-------------------' >> $HISTFILE
  cat $COMMANDFILE | dedup_history |
    sed -e 's/[0-9]*\.//' | tail -10 >> $HISTFILE
  history -c
  history -r
}

I personally did all this so that I'd have all my commandlines, forever, across all of my computers. So, clearly the default history size was far too small:


export HISTCONTROL="erasedups"
export HISTFILESIZE=10000
export HISTSIZE=10000

And that's a wrap! A simple script to copy the global history from each workstation up to a central repository and pull the workstation specific histories back to each workstation makes it cross-machine. For example, you can periodically copy ~/.histories/global to a central location as global.$HN, and copying global.* back into your history directory will get all the command lines from other workstations merged into your current environment.
I've had this scheme running for a few days, and love it! The idea that I now have a persistent store of all useful command lines may even change how I choose to write those command lines, making me deliberately include extra but useful information in them. Enjoy.

twitter's tweets

2008-03-21T23:06:00.000-07:00

So I signed up for twitter this weekend, and I have to say the jury is really out on this one. I mean, I can barely understand the huge appeal of blogging, RSS, and non-job social networking (myspace and facebook) — so why would I understand twitter?

In case you haven't heard of it, twitter.com lets you set up an account for broadcasting your mood. It's like IM status, without the IMs. All status all the time; all one way communication. Blogging for the severely ADD. Instead of getting a feed reader, you just set it up to text all your incoming streams to your phone, because you need more interruptions in your life to make you more efficient, right?

So perhaps I'm sounding so negative on it that there would appear to be no point to me even signing up. Except that's not so: what's interesting to me about twitter is that it creates a new kind of socializing; a new way to be connected with friends and family and strangers. Online personae are becoming something that more and more people have — it's not just a handful of geeks on IRC, or MOO, or tinyMUD, but rather a full-blown epidemic of artsy types finding ways to express themselves to a broader audience. In addition to the social implications, there's the possibility that an intelligent blogger in India could now earn a decent living from Google ads alone. Soon, this blogger may also need to twitter regularly, to remind her readers that she can be interesting in 140 characters or less as well as in longer articles. Twitter is becoming big enough to be part of an online persona that may someday be as important as having a nice shirt or a decent haircut: so it's time to sign up, and see how it works.

How it seems to work varies a lot according to who you follow. Some people twitter whatever silly thing crosses their mind. You won't follow them for long. Some people put so much thought and effort into deciding what clever thing they should tweet every 12 hours that you can spend minutes looking for the hidden meanings behind what they're saying. But I think most are just trying to be a little cool, and stay connected. And that's what the internet is all about.

twitter.com/osric

Truly on track again

2008-02-22T19:54:00.000-08:00

It's so good to be truly back on track - it's been going pretty well since my last post.

I did another closet cleanout. It was amazing how many shirts I had that just no longer fit - they're all too big. I wound up with a huge pile of large, xl, and even some mediums that had to go. It's so bizarre to clean out my closet and find nothing that's too small!

Finally, this time, a sweater that my daughter had picked out for me more than 5 years ago that has always been too small actually fit! It's a bit form-fitting, and last time I tried it on it fit but too well ... the man-boobs were just not tolerable ... but now I'm small enough that it's not really noticable and the sweater actually fits! Excitment.

An even slower fall!

2008-02-03T20:37:00.000-08:00

Well it is pretty sobering to look at the chart since my last post!

It is interesting to see what on the face of it is still progress (I am, after all, a little bit lighter) graphically shown to be not much progress at all. Too much travel, too many exceptions, punctuated by too little following the plan ... I'm sure when I get around to posting again things will look far better.

A slow summer

2007-08-18T19:43:00.000-07:00

I figure it's been long enough that I need another post about weight; and it's been slow going, for sure. However, overall I'm pretty happy that The Hacker's Diet continues to work for me, and also pleased to see its author provide a truly free web version of the required tools. I'm still using physicsdiet.com, because I like the prettier charts and can put up with the ads and the locked-in format. But I'm using The Hacker's Diet Online as a backup. I last posted on March 10th, and so here's the progress since then - most serious upswings are due to travel.

iPhone

2007-07-06T22:06:00.001-07:00

Well I hate a lot of decisions around the iPhone but that doesn't stop me wanting one. It just looks like one really cool device.

I'm glad someone else had the guts to take one apart.

Italic handwriting

2007-07-01T21:08:00.000-07:00

Truthfully, I'm at a loss to understand why we teach our children to write as we do. Traditional "ball-and-stick" letters are pretty ugly even when well-formed, and I know nobody who is proud of their printing or writing based on this foundation. Yet there must be arguments for it?

Academic arguments aside, there's the fact that my own daughter's writing is simply too messy for words. So I decided to teach her italics this summer—not to mention that I need to learn them for myself. I was surprised at how hard it was to find a good link on the subject on the web, but eventually landed at the ultimate destination, a small website that communicated in few words many things that I wanted to know. I recommend a visit; I cannot excuse my ignorance of the author until now, but can at least encourage you to read his text. http://briem.ismennt.is/

Stigma Schmigma

2007-03-10T11:34:00.000-08:00

Back in August, I posted about weight loss via reduced calories. While I still have a long way to go, I thought that it would be useful to record my progress to date.

The most obvious thing about this is that it's very cause-and-effect. I do need to get exercise, but I haven't been: so the only variable is what I eat. Eat less, lose weight. Eat more, gain weight. Who says you can't quit eating cold turkey? Sure you can.

Two frozen dinners a day (watch out for sodium - check out Nutrition Action's article) as much fruit and veggies as you want; cereal (hot or cold) for breakfast, and maybe one or two slices of bread. This simple, possibly non-nutritious eating plan can be stuck to - and you can tell when I've stuck to it, by looking for the downward slope on the physicsdiet.com chart.

More Eric Meyer on CSS

2007-01-24T23:27:00.000-08:00

Recently, I was in a bookstore and happened to see a copy of "More Eric Meyer on CSS". Of course, I had to have it ... everyone who does any CSS at all must know Eric's blog and cutting edge demonstrations and frankly, I owed him the price of the book for all I've learned for free.

The book has been pretty good, which is kind of disappointing. It has great step by step examples which help the reader get the feel of working through the examples themselves. Unfortunately I didn't open the book until I was on an airplane, and sure wished I'd downloaded the examples in advance.

The disappointing part was the last few chapters, especially on how to do menus and on how to style CSS Garden. They contain great content, but it's kind of old ... surely we've all seen how to do menus by now, and gone through the pages on how to do menus, and these chapters didn't really add much for me.

All that said, it's still time to say thanks. Buy one of Eric's books if you do CSS; you won't regret it.

How do you spell relief? e-m-p-t-y

2006-10-28T23:00:00.000-07:00

I finally convinced my wife today that we should try out rememberthemilk for our TODO items. We're already avid users of gmail and google calendar, and these tools are great for scheduling and basic communication, but there's a class of tasks that don't deserve a calendar spot, and don't really work well in e-mail, that just don't get done.

Along the way it seemed to me that we should probably buy a copy of Getting Things Done although I'm already rather a fan of
7 Habits et al. I also came across a handy PDF outlining how to manage your e-mail - namely keep your inbox empty. So I decided to apply these tools together and transferred my e-mail into rememberthemilk and was surprised - amazed - at the level of relief that created! Wow, there's a real risk here of getting satsified just by organizing the TODOs as opposed to actually doing them. At the same time, it's important to realize and accept that a big and growing inbox is in and of itself a source of stress; a suitable management system like GTD's 43 folders or rememberthemilk may help reduce that, if combined with an active effort to do the things in all those folders or on all those lists.

Certainly it's easier than ever before to get organized with tools that are freely available. As more and more software companies realize that they can make money by providing "free" or "almost free" services in your web browser, they've produced a huge range of choices - good, usable choices - for everything you need - e-mail, documents, spreadsheets, calendars, and TODOs. It's great! Now, I need to focus on actually doing some of those TODOs...

The Zen of RSS/Blogs

2006-10-06T22:54:00.000-07:00

I may, slowly, be beginning to understand the point of blogging and RSS as a pair. You'll have to bear with me if you've been into blogging from the beginning; I have to say that I personally was taken completely by surprise by the blogging craze and have had to work pretty hard to come up with something to post to this essentially unread blog every week or so.

Now, however, I'm starting to see it. The beauty of a blog post, as contrasted to e-mail or usenet (does anyone even remember usenet?) is that the author has to post with no real expectation of getting any responses. And the author is expected to post regularly; it seems that the real bloggers write multiple posts every day. Out of this mishmash arise a few blogs that are truly noteworthy; those end up in the RSS readers of the users ... it's a kind of double quality control. First of all the poster has to try to write something worthwhile or nobody will ever read it. Secondly, the RSS feeds that are good will eventuall show up where like minded people can find them, enabling a kind of monkey-see, monkey-do approach to quality control.

The feed reader interfaces are pretty good too. Not much challenge to guess which one I'm using, but there is plenty of choice and I'm sure everyone can find a favourite. So if, over time, I can fill my RSS reader with things I actually want to read, I will be left with the problem of finding time to read it...

greasemonkey rocks

2006-09-02T22:26:00.001-07:00

OK, I've been trying to get people interested in greasemonkey with absolutely no success. I don't get it! It's so clear to me that this is the greatest thing since firefox extensions ... all the more so because it's just so simple to get up and running (no "restart firefox to enable this extension" for each new user script).

The winner of the greasemonkey usefulness award has to go to the gmail keyboard shortcuts script. Don't just read this, check it out!

Dashboard ... pretty but just too slow

2006-09-02T20:52:00.002-07:00

So, I'm playing around with a new intel macbook and as part of settling into the machine started checking out dashboard widgets again.

I'd written off dashboard on my main Mac (dual 2Ghz PPC) because it's just too slow to start up, and too much of a RAM pig to be worth its weight. I thought that on the new much faster performing intel mac os, the situation might be different. So in fact I'm writing this blog post right from within Dashboard. However I'm afraid it'll be my first and last one -- because it's still just too slow to be usable. A real pity considering how beautiful it is, and the Google blogger widget is pretty cute, too (not that I'm biased).

Calorie Stigma?

2006-08-26T10:03:00.000-07:00

I am continuing to work on the Hacker's Diet and came across a livejournal log of someone who's used calorie counting to get down to about 128 lbs from 150ish. An interesting point they make is that they feel there's a stigma over calorie oriented diets, as opposed to the numerous "control what you eat" diets.

I found this really surprising and at the same time am realizing it might actually be true. Reactions to this (the latest of many weight control efforts on my part) has been less positive, and more focused on concern that it can't be done because of willpower requirements, or that it will lead to an unhealthy diet because there's no focus on what I eat. This makes no sense to me, because if normally I'm eating whatever I want (which I can assure you isn't all that healthy) and eating many more calories than I need, surely eating whatever I want but a reasonable number of calories is an improvement?

Looked at another way, the reason I'm so overweight is that my system is trying to be ready in case I should someday be starving. I have about 245,000 stored calories in my system - enough to support life on 2,000 calories per day for 122 days! So though it obviously wouldn't be good to starve for that long, the bottom line is that I could, in theory, not eat at all for four months and still survive!

The standard advice is to reduce your calorie intake by 500-1000 calories per day, depending on how obese you are. So for me, eating between 900 and 1500 calories per day is a completely reasonable approach. It's too early to say it is working, though the scale shows results, because it's the long-term outcome that matters. It's not to early to say, though, that eating less is probably required to lose weight and eating healthier and exercising is a good idea but probably not required to lose weight. So if you're struggling with recipes full of ingredients you've never heard of, trying foods you dislike, and reducing your calorie intake all at once ... it seems to me that you could do less, and get the result you're aiming for.

Kid Koding

2006-08-05T22:11:00.000-07:00

So my daughter has decided she'd like to start programming (with a little encouragement from me of course). We haven't actually started yet, but she has already made an elaborate plan: she wants to create a game for her brother. "In the corner of the screen will be a clock counting down two minutes, and there'll be hedgehogs, and he has to click on as many as he can before the time is up." Why hedgehogs? Don't ask such silly questions.

Today she elaborated a little further "I know there'll be six animals" (I had already suggested that we use different animals, perhaps ones from an existing game, since doing our own graphics would be hard work on top of writing her first program). "Daddy, how long will it take to write the program? Once I'm done, do I have it forever?"

It's very interesting to study her motivations and thoughts; as a child I came to programming anxious to duplicate the arcade games of the day. For her, it's all about entertaining her three year old brother, which is undoubtedly one of the favourite passtimes of my seven year old. We'll see how programming goes!

The Hacker's Diet

2006-07-31T17:01:00.000-07:00

The Hacker's Diet was recently recommended to me by a fellow coder who has lost a significant amount of weight on this program and kept it off for the long term. It's a hilarious read, well worth it ... and a sobering reminder that the only way to weigh less, is, in fact, to eat less...

Filled with good tools for tracking and correcting your progress, it's made me have a newfound taste for weight loss. There also seems to be a free web implementation at PhysicsDiet.com.

Visual Statistics

2006-03-15T19:35:00.000-08:00

As a long time visual thinker, I was very taken with the approach at gapminder.org, showcased at a recent tech talk given at Google (search for gapminder on Google video for the tech talk). Visualizing data in two or three dimensions can often give one powerful intuitions that justify following up with heavier-duty mathematics.
The basic purpose of visualization in this context is two-fold: first, to make a point in a striking and accessible way, and second, to allow you to focus on big trends that need further investigation. I love the approach taken during the video because it emphasizes the enormous disparity between the available data and possible methods of visualization and the amount of effort taken by the media to convey this information to individuals.
For myself, living in comfort in Canada, it's difficult to relate to a scale where the top end stops at $100 USD per day. That's 36,500 USD per year, and that salary level represents an absolutely infinitesimal portion of the world population. According to the CCSD, a family of four in Canada is below the poverty line if their income is below $35,455 CAD which these days is quite comparable to USD. The poverty line in Canada, in other words, is well above the level of the best incomes in most of the world (even after dividing by 4 to make the numbers directly comparable).
In any case, don't look at my numbers. Watch the video and visit gapminder.org ... I can almost guarantee you'll learn something new.

ALXN bounces back

2006-03-07T09:08:00.000-08:00

I was stopped out of this position the day after my last post by an over-aggressive stop, but it looks like it was a short-lived idea anyway (pun intended :-)

ALXN down from it's high?

2006-03-02T19:48:00.000-08:00

I have to say I envy those who didn't lose faith when ALXN fell from $28 down to $20, because just a few days ago it was trading at around $38, a very nice 1.5x for anyone who saw the recovery from $20 and jumped on board.
Sadly, I'm not one of those people, and I'm here to celebrate the fact that the party's over - ALXN's down to $35.25 after hours. I'm short - are you?

Great Games with Marginal Websites

2006-02-28T20:14:00.000-08:00

A key part of my philosophy is that it is important to stretch your mind in a lot of ways. Just like a runner could justify swimming, weight training, and other fitness activities in order to maintain total body fitness, someone whose primary asset is their brain should use it for lots of different thinking tasks.

With that in mind I'm often looking for novel games, and over the years have come to appreciate certain games and/or certain publishers. Today I want to focus on two, which used to have something in common: a great game with a marginal website.

The first of these is Tantrix, whose physical game tiles are almost impossible to find in stores except for the tiny 10-pack. This is a great game which has been out for years, and in 2000 it had a website which looked like it was designed in 1993. Now, however, the website is looking more modern and professional, and I wonder if it's made a difference to the adoption of the game. In any event, it's well worth a visit, especially to try your hand at playing it online. It's what I'd call a traditional partial-information strategy game.

The second of these is Set, a novel card game which is easier to find in stores but whose website looks like it arrived at the same time as Mosaic (for those who don't know when that was, we're talking before there was any real content on the WWW). Nonetheless Set is a unique game in that players of all ages can play on an even playing field, because the ability to do the pattern match is about as well established in 9 year olds as it is in 40 year olds. As such, it's a unique game for parents who can play their best against children, with the children still having a good shot at winning. It's also easy to play in teams if you have children younger than 9 in the game.

Working at Google

2006-02-28T20:10:00.000-08:00

On Monday, I will fly to Mountain View to become oriented as a "Googler". I will be the fourth employee in the Waterloo, Ontario office and we're certainly looking for people so please send your resume through to Google via their standard channels. Please don't e-mail me about positions at Google unless you know me, or know someone who knows me; I will not recommend you unless I actually know you at least second-hand.

[Repost from Feb 11, 2006]