Planet Autopackage

Jan Niklas Hasse: Google Mail's Anti-Spam Filter becoming stupid?

Thu, 04 Mar 2010 18:17:12 +0000

Today I received two emails which where spam. But it was so obvious. They only contained images (!), where sent by very cryptic addresses, and told me how I could save money/taxes (in the title, I didn't view the images). I don't know what the hell Google is doing.

Jan Niklas Hasse: LinuxTag 24-27 June 2009

Wed, 17 Jun 2009 21:39:48 +0000

Autopackage (me and a friend of mine) will be in hall 7.2b booth 101b. I will post some more news on the mailing list soon.

Isak Savo: Perl was born

Fri, 15 May 2009 19:57:08 +0000

"1987 - Larry Wall falls asleep and hits Larry Wall's forehead on the keyboard. Upon waking Larry Wall decides that the string of characters on Larry Wall's monitor isn't random but an example program in a programming language that God wants His prophet, Larry Wall, to design. Perl is born."

Or why not

"1995 - Brendan Eich reads up on every mistake ever made in designing a programming language, invents a few more, and creates LiveScript. Later, in an effort to cash in on the popularity of Java the language is renamed JavaScript. Later still, in an effort to cash in on the popularity of skin diseases the language is renamed ECMAScript."

For more fun, read A Brief, Incomplete, and Mostly Wrong History of Programming Languages.

Isak Savo: Disconnected from reality?

Fri, 01 May 2009 18:25:33 +0000

(Warning: this post is just an angry rant, feel free to skip it)

I finally got sick of Ubuntu (lets see how long that lasts) and wiped by root partition to put fedora 11 preview on it. The first thing I always do on any linux distribution is to install support for all the non-free stuff like mpeg codecs and nvidia drivers. A quick googling on "fedora 11 restricted formats" gave me this page: http://fedoraproject.org/wiki/ForbiddenItems

Are they out of their f***** mind? I agree in principle to the stuff on that page, but in the real world, that page is just rubbish.

I have an nvidia card (and a good one) and lots of music in mp3 format (not every device or player in the world speaks ogg) and movies in a variety of formats. I would love to live in a free world, with all free formats and no software patents, but were not there yet and somehow I doubt we will ever get there.

To me, that page is a slap in the face:
"Proprietary drivers are not included in Fedora. They are considered harmful by many kernel developers."
I read: blabla, bla bla bla. Bla bla bla. You stupid user, you choose the wrong hardware. Now fuck off.

Nuclear waste is considered harmful. Proprietary drivers is at most stupid

PS. I know I will be able to get everything to work eventually. This rant is about the attitude of that page.

Update: http://rpmfusion.org/Configuration/ and http://rpm.livna.org/ seems to contain what I need to be able to use my computer again.
Update2: I'm back on Ubuntu again. Too bad, because I think the work red hat employed hackers are doing with free software is awesome and it would be nice to run the distribution where it happens.. well well, the goodies end up in ubuntu at some point anyway.

Jan Niklas Hasse: Autopackage @ LinuxTag in Berlin

Thu, 30 Apr 2009 14:55:58 +0000

I've just recieved an E-Mail that Autopackage qualified for a sponsored booth at this year's LinuxTag. I'm not sure if that means that we already got one for sure though. Anyway, I will have to start working on completing the project's profile.

Unfortunately I'm still busy with school leaving examination for secondary school, but I'll be finished May the 13th. My current plans are to work a little bit on apbuild since the upcoming release of glibc 2.10.

Isak Savo: First shot at synchronizing with the iPhone

Sat, 18 Apr 2009 16:15:34 +0000

As I mentioned in my previous blog post, I'm currently a proud owner of an iPhone and needed a way to synchronize my Lotus Notes calendar with it.

I'm not a big fan of CompanionLink, since it is both expensive (50 bucks) and pretty buggy. It also doesn't sync everything the way I want (more on this further down). Apart from companionlink, there are no tools out there that can help me get my calendar to the iphone (without going through a 3rd party) Being a software developer, and driven by interesting code challenges, the obvious solution to this problem is of course: to write a tool myself. :-)

Introducing: TieCal Synchronizer, your one-stop tool to synchronize Lotus Notes to the iPhone. It's still very rough, and has some annoying limitations (like no support for repeating events) but it works fairly well for me. It only supports one way sync (this is mostly all I need anyway) and lacks some basic configuration options, like setting timespan, and turning on/off reminders. It has also never been tested outside my own setup, so your milage may vary...

What it does do that you can't get with CompanionLink is merging of Lotus Notes' "Room" and "Location" fields so that they appear on the iPhone. Outlook only have the Location field, so Room is silently ignored by CompanionLink which is very annoying when you're on your way to a meeting and forgot to check where it is... TieCal handles this properly

The code is open source (GPL v2) so feel free to try it out. It's written in C#, using WPF for GUI. I've not yet created a proper release, do you'll have to compile from source for now (VS2008 needed).

Isak Savo: Stuff they don't tell you about the iPhone

Mon, 13 Apr 2009 12:38:14 +0000

I bought myself an iphone the other week. Yeah I know it can't do MMS and that the battery life sucks but it's awesomeness of the user interface and amount of fun and useful stuff in the app store more than compensates for those issues. Besides, the crap that comes out of both Nokia and Sony Ericson these days are so pathetic that it's not even fun anymore. Come on, Nokia hasn't done a useful phone since the 5110 and I can't remember (Sony) Ericson ever producing a useful phone...

However, there are a few quirks you should be aware of that they don't tell you when you buy an iPhone. These are things that came as a surprise to me after I've bought it

The "universal" dock
Apple completely screwed up the dock connector thing. My initial thought was "Awesome, since the iPhone has this dock connector, I should be able to use all the sweet iPod accessories out there, right?" - WRONG! While they are all the same physically (the device will fit any ipod cord or accessory) you are not guaranteed that it will work. Apparently, apple has changed the interface so many times now that it's sheer luck if you get it to work. I did a quick unscientific test at the local Media Markt store and tried to put my iphone into the 20-something different ipod compatible devices and none of them worked with my iPhone 3G. A handful of them did at least charge the phone but not even that is guaranteed.

USB compatibility?
I talked to a clerk at another store when I wanted to buy a car charger and he confirmed this problem. Apparently it's even worse that this. Even if you have your standard iPhone usb-to-iPhone cord (from apple, shipped with the phone) and connect it to a third party wall-outlet-to-usb you cannot be sure it works. The clerk said they had two different ones in store since they work on different iPod/iPhone models. USB has been around for years and it's not rocket science. There's 5V on pin 1 and ground on pin 5 and you can draw up to 0.5A on a single port. How hard could it be? There's no room for screwups, but they still managed somehow.. *sigh*

Getting data to and from it
While we're at it, let's whine some more at Apple :). To do anything with your iPhone from a computer, you will need to go through iTunes. Anything except photos, which seems to be using some sort of standard protocol (ptp?) iTunes, apart from being a pretty crappy music player, only runs on Windows and OS X, which is of course problem for Linux users. I'm using windows at work and also have a Vista partition on my home machine so it doesn't bother me that much. Also, I'm not using the iphone for music (I have the far more superior Cowon D2 for that) so I'm not hindered by having to use iTunes for my music either. The thing I do have a problem with is synchronizing my calendar and contacts. At work, we use Lotus Notes which isn't supported by Apple (Nokia and Sony Ericson both do, by the way) and the only way to do it is to use a third party product (CompanionLink, $49.99) which is not only expensive, but it's also really crappy (random hanging, lack of decent configuration). It's also sort of a hack since it goes through Outlook before putting data on the iphone (itunes only supports Outlook and as I mentioned you have to go through itunes to get data on the iphone). I've no problem with this approach per se, but I object to putting out $49.99 for a buggy hack just because it happened to be the only option available.

We haven't seen the end of this story yet, but I plan on writing my own (hackish) sync solution that does what CompanionLink does and then release it as open source. It will do a bare minimum required to let me view my calendar on the iphone - no advanced two way merging or stuff like. Now let's see if I succeed..
The status right now, after one evening of coding, is that I can read calendar entries from both Outlook 2007 and Lotus Notes 7. Now I just have to write the syncronization code and then use itunes to put it on the phone. Should be easy, right? :-)

Despite the above, i'm satisfied with the iPhone. It's an awesome piece of technology that simply looks astonishing...

Mike Hearn: mikehearn

Tue, 24 Mar 2009 18:17:13 +0000

I just uploaded my first Android app to the market, Facebook Sync for Android. It is very simple – it just copies your friends photos into your address book, so you see their faces when they call you. Right now that’s about all it does, but I’m hoping I can make it significantly more useful in future.

So, how was my first time?

Mostly, writing Android apps is easy and fun. Occasionally, it’s frustrating and annoying. The biggest problems I had were caused by bad documentation, buggy tools and inconsistent UI in the included apps making it hard to know what to aim for. I also wasted significant time on the RPC framework which turned out to be unnecessary – hopefully I’ll be able to submit a documentation patch soon that clarifies when you need to use it and when you don’t.

That said, Android is about a million times nicer than J2ME. How it compares to the iPhone is the real question, and I haven’t written an iPhone app so can’t do a real comparison there. I will say that I’ll put up with significant pain to get Java over Objective-C, which as far as I can tell is a stupid language with no merits at all. And I’m not a Java fan, so that’s a pretty strong statement for me.

I’ll be interested to see how much the app is used. Right now the only place contact photos are used in Android is the incoming call screen. Once you pick up, it’s gone. If you call them, it doesn’t appear. The text messaging app doesn’t use photos at all, and so on. I think this is a chicken and egg situation …. nobody is going to devote precious screen space in their app to a photo box when most people won’t have any contact photos. If syncing from social networks becomes widespread, it’ll suddenly become worthwhile to use photos more often, which I think will result in a much nicer experience.

Jan Niklas Hasse: Autopackage 1.4.0

Wed, 04 Mar 2009 18:34:06 +0000

I couldn't wait any longer and released Autopackage 1.4.0 today. Hope everything works fine, please report any bugs left. There could be some things I forgot or did wrong since it's my first real release.

Thanks for all the feedback and support and also many thanks to Isak for helping me out with things like CVS access!

Isak Savo: Living with multiple operating systems

Wed, 25 Feb 2009 20:31:32 +0000

My old computer died last year just before christmas so I had to buy a new one. I bought myself an Acer Aspire M5641 which is a decent machine with a decent price tag.

It came pre-installed with Windows Vista, which I hadn't tried until then so I decided to give it a try. It had a solid 5.0 on the experience index so I could test all the shiny effects that's supposed to give you the WOW effect.

All in all, it's a nice improvement over XP, but if you're coming from a Compiz powered desktop, or have seen OS X, then there's definitely no WOW what-so-ever.

The UAC thing was not as annoying as I had expected. It's a bit more frequent than the average sudo prompt in ubuntu but not at all as annoying as various reviewers had described it.

What finally made me plug in another hard disk and install Linux again was the lack of a decent command prompt. You can't really appreciate how much you miss it unless you spend a couple of months on a Windows system. I tried to learn the powershell thing, but after two days trying (and failing!) to create the equivalent of the snippet below, I gave up.


for i in *.avi; do
    output=`basename $i`-converted.avi
    RunCommandToConvert $i $output
    RunCommandToFixConvertedFile $output
done

Passing CLR objects around may sound nice in theory, but it's freakin impossible to use when all you want to do is manipulate strings and do operations on various files. Maybe my use case is not within the scope of powershell, I don't know. Or maybe it's just a case of trying to learn an old dog new tricks...

In any case, I'm now running Ubuntu 8.10 alongside windows (I decided to keep vista around, I mean I've payed my microsoft tax and it's useful for certain things like playing games or talking to weird hardware) and that requires sharing of data between two operating systems.

Most important is:

Pictures
Music
Firefox bookmarks and passwords.

Thanks to various software, I can access ntfs from Linux, and ext2 from windows. Both reading and writing works fine.

For pictures, this was easy. I'm using Picasa (it's fantastic! And I couldn't care less that it isn't "native" or anything.) and since my photos are stored in separate folders, and picasa stores settings for these in its picasa.ini file in those folders it's just a matter of copying over the files. Star rating, comments, edits. Everything already migrated with a simple 'cp /windows/picasa-folder ~/Pictures'. The only thing missing is the "Albums" which are sort of virtual grouping of pictures. I will need to migrate these manually from windows somehow.
For keeping these two folders in sync, I'm using Conduit which will automatically sync the folders for me.

For Firefox, I used to use Google Browser Sync, but that has long since been abandoned. A quick googling found me the Foxmarks service which so far has worked great. It syncs bookmarks and passwords (not cookies or preferences) but that's the most important things anyway.

For Music, I just made a symbolic link to my mounted windows partition. In windows, I used Winamp to listen to music (it's the player that sucks less) and on Linux I use Banshee. The problem with this approach is that neither of these players store all their information in the actual ID3 tags of the files. Most of the information is there, like album, title, artists etc. But I also like to rate my songs (to use the "Highest Rated" automatic playlist) as well as look at the play count (for fun, or use the "Most Played" automatic playlist). This information is typically stored in each applications own database, in their own format.

To solve this, I'm working on a small tool to migrate all this secret information between the players. I will save the details for another blog post, but right now I'm focusing on getting information out of Winamp and after that, getting extracted information into Banshee.

Isak Savo: úâ¶¯ Þ3!Ú¦Ø§ú0z·o"Ú²×æ

Sat, 17 Jan 2009 09:53:38 +0000

Time to travel again, but this time for pleasure instead of business. I'm going to Argentina with my family to visit my brother who's studying there at the moment.

Trip will include (but not limited to)

A taste of the argentinian kitchen. Especially the meat which has a near-legendary reputation.
A few days exploring Buenos Aires
A trip to the waterfalls in Iguau
Visit to the wine districts in the west, around the city of Mendoza

It's gonna be awesome. Haven't seen my brother since this summer so it's gonna be fun see him again. I will bring all my camera gear and I'm sure there's gonna be some great photo opportunities.

Jan Niklas Hasse: Autopackage 1.2.6

Tue, 13 Jan 2009 22:19:13 +0000

Unfortunately 1.4.0 still isn't ready to be released, but Isak released 1.2.6 yesterday.

Also the mailing list has got quite active in the last few days, I'm surprised.
Hopefully I will find some time to work on 1.4.0 this weekend.

Jan Niklas Hasse: Autopackage 1.4 progress

Sat, 29 Nov 2008 19:34:48 +0000

It has again been a long time since my last blog post. Well, all I can say is that nearly all things are finished for 1.4 (3 tickets left in trac).

The only thing to do now for me is testing it on many distributions. With no more exams and the upcoming Christmas holidays this should be done in a few weeks and I hope we can release 1.4 this year.

Mike Hearn: mikehearn

Sun, 23 Nov 2008 16:49:28 +0000

I had forgotten what a giant pain in the ass web design can be. That is, I had forgotten until last week, when I started compatibility testing a website I made for a friend who’s starting up a new business.

BP-Schulz is a one-woman company in Zurich specialising in hypnotherapy, counselling, relaxation and other things. I’ve been friends with Nastassja for about a year now so creating a web site (and a logo, and business cards) was a good way to help her get established.

We started with a graphics design session, and settled on a combination of her color scheme and my jigsaw puzzle motif. If I’d known how awkward that’d be I might have chosen something different. Making it look OK in Firefox wasn’t too hard, although I resorted to JavaScript absolute positioning to make the extrusions and holes in the pieces (ew). But in Internet Explorer I had to wrestle with the nightmare of “layout”.

It turns out that Internet Explorer has this imaginary concept of elements having “layout”. Layout isn’t something you’ll find in any spec – and for the longest time, not even in Microsofts documentation. Instead it’s an implementation detail of how Trident, the IE rendering engine, works. Elements that have layout are responsible for arranging their child subtree, elements that don’t are arranged by their nearest parent that does. Sounds simple? Unfortunately not.

Almost any element can be made to have layout, but only indirectly. There is no layout = true property you can set. Instead boxes get layout implicitly by setting width or height on them, or floating them, or being generated from a pre-set list of elements that always have layout. Whether an element has layout or not affects how it interacts with floats, how it positions, how it sizes itself and almost anything else you can imagine.

The only way to stay sane whilst developing non-trivial CSS on Internet Explorer is to achieve a zen-like insight into the mysteries of layout. Only once you understand the fundamental architecture of the browser itself can you create web pages that look (almost) how you wanted them to.

So do yourself a favor – go read the article I linked. Think about every sentence hard, as it’s easy to become lost in a maze of floats, boxes, containers and elements. Then go back and fix the websites you never got around to fixing.

Isak Savo: Back from PDC

Sat, 01 Nov 2008 16:48:46 +0000

Came back from my trip to LA yesterday and I must say it was a blast. I've uploaded some photos to my picasa web album, click the collage below to get to it:

From PDC08

The big announcement this year was the new Windows Azure Cloud service that I have mixed feelings about. I think it could be useful for small businesses and startups to offload the maintenance of datacenters to Microsoft (and having them deal with scaling issues), but I doubt enterprises would like to have microsoft handle all their presious data.
They also showed alot of the upcoming Windows 7, which in my (and many other people I talked to during the week) opinion is more like "Vista as it was supposed to be" than a radically new version of Windows. They finally have the new search stuff in, including support for virtual folders to replace the "My Documents" / "My Photos" stuff that have been around since the '95 days. In Win7, you have "libraries" instead, which contains an aggregated view of several folders, so for example the "Music Library" could contain all files in $userhome/my music, d:\music, e:\other-music and still be maintained from what to the user seems like a single location. Nothing radically new (the idea has floated around the OSS community for some time) but I think it's nice nontheless. If nothing else, it makes it easier to separate the user data (docs, music, videos, etc) from the system data (os files, program files etc), something which has been possible but quite hard before in windows. (In Linux, of course, you just mount /home from a separate partition and be done with it)

The big thing about win7 in my opinion will be the addition of multi touch devices as a first class input device. This means that if you have a multi touch screen, like the newer displays from HP) they will seemlessly work with windows 7. In the open source world, multi pointer x should be able to provide the same kind of built-in functionality and I hope distributors will begin to incorporate ones it matures a bit more. I'm definitely keeping my eye on that project.

I also got a chance to talk to the Novell people at PDC. Aaron Bockover from the Banshee project was there and he showed some of the cool stuff he's been working on for banshee, including the d-bus interfaces and port to OSX. I asked him about what he thought of adding full WPF support to mono and he gave the "we've talked about it, we're interested in it, but we don't have any plans to do it yet" answer. Unfortunately Miguel was never around the booth when I was there so I never got the chance to ask him directly about it. I did go to see his talk about Mono and .NET and it was one of the best talks of the entire week (and that says alot, since there were many interesting sessions there!). He showed the new C# command prompt thing which was received with apploads and laughter since just a few days before, Anders Hejlsberg (chief architect for c#) showed something very similar but that is just in pre-planning and won't be shipped for several years (if at all). Nice to see OSS is taking the lead here.
Miguel also talked about the move to mono done by many game vendors and how the open model of mono enables them to use it in ways that simply wouldn't be possible using microsoft's closed license model. Game logic is traditionally written in slow scripting languages like LUA to make it easier, but it also makes it less performant. Using c# instead seems to provide a very good middle ground for these kind of things.

Update: direkt link to miguel's talk.(wmv download. Also available as mp4 download)
Update2: fixed mp4 link

Isak Savo: Attending PDC08 in LA

Thu, 23 Oct 2008 16:17:27 +0000

On Sunday, I'm leaving for Los Angeles to attend Microsoft's PDC 08.

Looking good!

One talk I'm looking forward to slightly more than the other talks is Miguel de Icaza's talk about Mono and .NET. I secretly hope that Novel will announce that they will bring us full WPF support instead of just the limited subset called Silverlight. We're already starting to see similar platforms (like Clutter) in the open source world and wrapping this into WPF for Mono sounds like a nice thing to do. No idea if it's feasible or even possible though :-)

Isak Savo: Do you really need touch screen?

Thu, 16 Oct 2008 20:42:44 +0000

Are you sure you absolutely need a touch screen on you cell phone? If not, then there are alternatives to the iPhone:

I guess the moral is that you shouldn't pay too much attention to the specifications, or at least that they don't convey the full story.
Or maybe they are saying that the iPhone is just marginally better than a rock. Who knows :)

Isak Savo: Humanity

Tue, 14 Oct 2008 18:26:14 +0000

This is old news, but I haven't heard about it until today when we were at the supermarket:

Not completely unlike the other famous ubuntu, it tries to live up to the meaning of the word by being fairtrade certified. The taste, however, wasn't all that good - it's drinkable, but it's not like a real coke.

Neil Munro: The road to 0.1.6

Wed, 24 Sep 2008 01:27:39 +0000

Since my last post I have been hard at work, since I passed my exams and what not I made it into my third year of Uni, which since I didn't have to worry about them too much anymore I could get on with warp, now I have done a LOT I shall document a few changes.

1) Re wrote Add_Feed, Edit_Feed and Remove_Feed in cli.cs, they're simpler now, don't have the user double and triple checking the feed, in a gui once ok is clicked it's saved, the command line interface should maintain this consistency.

2) I have made warp much easier to work with if you wanna create a new UI you only have to write your file and create the relevant entries in UI.cs.

3) 4 bugs fixed TODAY, two instances where null feeds could be added and two instances where URLs containing spaces could be added.

4) vRunArgs has also been re-written and simplified.

5) A bug where the user was asked to press any key to exit, but the user additionally had to press return to exit, so a Console.ReadLine( ) was changed to a Console.ReadKey( )

6) A whole new preferences system was written, feeds and preferences are all stored together in one easy xml config file (I say easy, it certinally wasn't easy to write!), it's also stored in a logical place on the hard drive depending on platform (although it's currently probably incorrect under windows!).

7) A database will be implemented for 0.1.6 to managed the GUID's since any other method could result in variable's becoming overloaded, a database should be a bit overkill at the moment, but it'll future proof warp.

8) All funtions in Application.cs set a flag in the object and save the error message if an exception is thrown, a function exists to grab the error in question, this enables ANY UI to grab the error message and print it to the user.

9) The FeedManagementV2.cs file has been removed since Application does everything and more than it ever did.

10) Warp now has to be run as the root user, you know, since it's kinda a root application, I have yet to implement similar functionality under windows, since I'm not entirely sure how windows handles user privelages.

There we are, top ten things I've been working on, the database is the next thing, since a way to search and retrieve GUID's is essential for update and upgrade!

Blogged with the Flock Browser

Isak Savo: New programming site

Sat, 20 Sep 2008 10:19:01 +0000

The famous (in the programming world at least) Joel Spolsky (Joel on Software) and Jeff Atwood (Coding Horror) has created a new kind of programming resource web site called Stack Overflow. The site is sort of a wikipedia meets Experts-Exchange meets digg, with content licensed under the creative commons. Of course, I signed up the day the site entered public beta, and I'm working on building up my reputation on the site.

The reputation system is what makes the site especially interesting, because it allows the site to be run by the community without the explicit need to assign moderators that control the content. Anyone with sufficiently high reputation can do those kind of tasks. And getting a high rep is not very easy - or at least it takes time and dedication. As I said, you gain rep by answering questions, but the thing is you only get it if other people like your answers, giving you an incentive to write as good and detailed as possible. And only people with a certain amount of rep can actually vote, so you can't easilly trick the system by creating a bunch of fake accounts. Really clever.

There's more info about the reputation system and the site as a whole on the FAQ page. Also, Joel Spolsky's post about the launch explains a lot too.

Neil Munro: Title

Thu, 18 Sep 2008 18:21:43 +0000

I've been thinking in addition to fixing bugs for each release I should not wander directionless into the coding wild, I need to have a destination, so since I think that this is such a great idea I am focusing on important tasks for each release. For now the security policy code is going to be ignore, only for the time being, it WILL be brought back in.

0.1.6 (the next stable release):
This will not be a total re-write, but a LOT of the code is going to be tidied up and cleaned to it's not a horrible mess like it previously was.

0.1.7 (the next unstable release):
I've been thinking, I could REALLY make my life easier than support hundreds of different package managers, I could just support packagekit, I make ONE package kit call and the package is installed, I don't even need to know what distro is being used! PackageKit solves a lot of the possible problems I may run into so since distributions are packaging it now and the next bunch of distros will have it installed or available in a repo Warp will begin to depend on packagekit to do the muscle work for distribution specific packages. Admittedly source packages will need to be handled by warp, but this is fine since I can focus on that and rely on packagekit.

0.1.8:
PackageKit integration will be stabilised if it's not already, hopefully the newly organised code from 0.1.6 will be good enough for a long time now and I can avoid tangling it all up again. If this is the case, then hopefully packages can be uninstalled, at the very least, through packagekit, autopackage may be another story.

0.1.9:
Windows features will be focused on this release, I can only do so much under linux that will ensure windows integration, there will come a point where I simply just have to boot windows and get on with it. Also with that, a System.Drawing based GUI can be implemented the Mono/Wine would be able to run, not Linux native but it should run :D

0.2.0:
If at this point all core functions are done and a rudimentary GUI runs on at least windows, then Warp will ship 0.2.0, work will the begin on reimplementing security policy and Linux native GUI'S.

Blogged with the Flock Browser

Jan Niklas Hasse: GtkMencoder's Trac page and hybrid autopackages

Wed, 17 Sep 2008 22:18:53 +0000

It has been a long time since my first post, so here's the second one.

First I want to present GtkMencoder, a frontend to convert video files using mencoder. I started it last year because I didn't want to look up mencoder's command line option all the time. It was also a good time to practise python as it's written using PyGTK.
Two friends of mine helped me set up a Trac page. Have a look and tell me what you think of it :)

I also committed some changes to autopackage that should make it possible to create hybrid packages. This means you can create a package, which installs different binaries on 32 and 64 bit machines using binary patches (similar to the already existing C++ ABI mechanism).
To create such a package you first need to create a package which only works on one platform, let's say x86. Then you copy this package to your x86_64 machine and let makepackage know that it should create binary patches. For this you need to add the following lines to the BuildPrepare section:

if [[ "$(uname -m)" == "x86_64" ]]; then
    export HYBRID_PACKAGE="your_x86_only.package"
fi

makepackage will extract the package and use its data payload to see what differences there are. The newly created package should then be installable on both 32 and 64 bit machines. Depending on the target machine autopackage will install the right binaries.

There are still some bugs left, but I hope I can fix them during the next two weeks so that autopackage 1.3.0 can be released and you can help us finding bugs.

Neil Munro: Progress on 0.1.6.

Mon, 08 Sep 2008 01:20:52 +0000

Despite me being reluctant to continue warp development with the inability to upload all my changes to an svn server I have continued to work on what will become 0.1.6, many many features present up to now have been disabled in order to implement the preferences code, which is a bit misleading, while it WILL be an object containing all the applications preferences, it'll also contain data critical for the successful running of warp, as is stands objects are being passed all over the place and it's an absolute mess, really it is.

My central starting point is the loading of the program data, which will be in charge of what UI to load etc, it'll also abstract things away from a specific implementation, or at least, that's my plan, so far little or no code has been thrown away, but I am simplfying the code, for example:

I now check the platform and set up paths reletive to each OS, this enables me to get the paths once and then use the varibales anywhere in the program without rechecking the OS.

It was probably insanity of me to not do this in the first place, but I'm working on it now.

Currently, for some reason I have a sharing violation which is holding me back and that I am trying to track down, I may just start a new project an just use the new code see if it's my new code or the old conflicting with it, well that done, and it's the new code not working already :P But, messing around with the new code in a new project enabled me to fix the issus and merge the changes back into the old project, which is a step forward. :D

Blogged with the Flock Browser

Neil Munro: Update

Thu, 04 Sep 2008 22:32:18 +0000

Warp was on an undefined but not permanent hiatus, a family member died and my best friends father also died, uni exam results were not what I was expecting so work on Warp was put on hold until things quieted down, fortunately I am back, however the sourceforge svn server isn't up for write operations yet, so it'll be a while before any work is committed.

I have been thinking that the state of the code has become quite hard to manage, so I am looking for ways to streamline the code in order to make any further progress with it.

I also note a complete preferences file is required, in fact, feeds and filters could be put into the one configuration file and prevent files being scattered all over the place.

So in an attempt to prevent code being chucked out, since it's known coders dislike throwing away code, I am going to see if I can re-structure before it's too late...

Blogged with the Flock Browser

Mike Hearn: mikehearn

Wed, 09 Jul 2008 17:33:13 +0000

… is growing slowly, but most people still aren’t making the connection between high oil prices and the fundamental shift in production rates that is peak oil. This image shows Google searches for “oil price” (red) against “peak oil” (blue):

Mike Hearn: mikehearn

Mon, 07 Jul 2008 22:51:06 +0000

Yay, it finally happened. Google released protocol buffers as an open source library.

Protocol buffers are one of those ideas that has a million different implementations but very few good implementations. As a result everybody invents their own and we lose interopability and time for no good reason. Protocol buffers more or less hit the sweet spot between features and simplicity – I certainly wouldn’t want to design a file format or network protocol without them these days, and now I won’t have to, even if I’m doing work outside of Google.

Here’s a quick intro to what they are.

Protocol buffers are binary XML

Well, sort of. XML is more complicated than protocol buffers are – protobufs have no concept of namespaces, attributes vs subelements, character escaping, DTDs or any of the other things that make XML complicated. But the essential idea is the same – it’s a way of representing trees of structured data in such a way that they can be extended in a backwards compatible manner.

The key features of protocol buffers are:

A very efficient yet simple binary encoding. A minimal protobuf takes only a few bytes due to clever use of variable-length integers.
A simple specification syntax that lets you define a schema far more easily than an XML Schema would be.
A compiler that produces objects representing your structures in either C++, Python or Java. These objects present a much cleaner and simpler interface than the XML DOM – it’s quite feasible to represent all your programs internal state this way, whereas it’d be painful to replace a native object heirarchy with XML.
A set of tools that let you quickly serialize and deserialize them.
A lightweight (incomplete) set of interfaces that let you hook protobufs up to an RPC system.

Usage in RPC

Protocol buffers are so named because they were developed as a wire protocol for server communication within Google. Over time this developed into a full high performance RPC system that supports many advanced features. In particular it’s very easy to debug and troubleshoot, a feature I find invaluable day in, day out. At heart though, this RPC system is still based on protocol buffers – the RPC protocols are defined using them.

The key feature here is that it’s very easy to extend protocols based on protobufs over time. Alternatives like CORBA, DCOM or Ice don’t have a particularly elegant approach to this – if you want to introduce a new parameter to an RPC for instance, you need to introduce a whole new interface, and then translate the call through. In a protobuf based system, you just mark the new field as optional and then use the has_foo() function before accessing it in your new server. When the client is ready to use the new parameter, one set_foo() call is all it takes.

The next key feature is that it’s very efficient on the wire. Protocols like SOAP or XML-RPC are really not designed for efficiency at all. In many cases, this won’t be a big deal, but for Google it is because we push everything so hard.

Protocol buffers as a file format

Because they efficiently serialize to binary, can be used from at least 3 key languages (and more in future) and can be extended over time, protocol buffers are a perfect fit for many file formats. Most open source programs these days have based their file formats on XML. OpenOffice, AbiWord, Inkscape and more all use XML markup languages to save their data. Because XML tends to be very large, they often zip it, resulting in a very slow and complicated piece of code to load or save these files. This matters – a big part of the complexity of the Word/Excel formats is due to “quicksave”, a feature that users love as it lets auto-save be less intrusive and more frequent, but which complicates the codebase considerably.

Protocol buffers have a more or less optimal binary representation and can be deserialized into in-memory objects extremely fast. Nothing stops you gzipping part or all of them if you want to eliminate some redundancy, but it’ll be redundancy on the application level that is eliminated, not on the format level.

Go check it out

Seriously, it’ll only take a few minutes to read the docs, and you’ll add a valuable tool to your arsenal. Whilst the problem that protocol buffers solve isn’t new, this is one of the best implementations I’ve seen so far. I hope the open source community embraces this system as a way to make easier to use, more efficient file formats and network protocols for its applications.

Neil Munro: Protection charms.

Thu, 19 Jun 2008 00:35:20 +0000

When I was in Turkey I was introduced to a local legend, that people with blue eyes and blonde hair had the power to curse people, usually accidentally, now being blue eyed and blonde haired, I was made an example of during a demonstration of the legend, but I met someone randomly on the bus yesterday who was wearing the 'evil eye charm' and this is supposed to repel the ability to curse someone. It really is hilarious to walk down the street and see people wearing something that is believed to render my ability to curse someone useless.

Not that I have tried to curse anyone, but it's hilarious and interesting all at once.

In other news, warp is coming close to being autopackagable, once I have the know how to package properly I just ensure binary relocatability remains in place with new code and it's another feature completed on the way to 0.1.6.

My friend Ash will be visiting from Glasgow in a couple of weeks, and she will be doing one of the minor features for 0.1.6 not that I can't do it, but I wish to encourage people to work on Warp. She will be implementing the creation of the feeds.txt file and adding of the default feed if it is not present. This will mark another bug/feature (I cannot remember what it is labeled as right now) complete for 0.1.6.

Work is progressing slow but steadily. :)

Blogged with the Flock Browser

Neil Munro: 0.1.6 progress

Mon, 16 Jun 2008 14:33:28 +0000

Hard drive failure, PC World, blah blah blah.

Restored my system, since I am getting quite good at that now. Continued to work on Warp.

The Warp autopackage builds, and installs but does not work, yet.

9 of the 17 tasks currently marked to complete have been completed, putting progress at over 50% so far. So far completed is

1) All the splicers are now in their own sealed class, help has been moved to a sealed class.
2) All code now has a more complete style.
3) All functions recurse if they fail, hopefully this will mean something unexpected will be attempted again, it may lead to infinite loops, but this remains to be seen.
4) Bug fixes etc
5) The clean argument now works. This had not been added to the bug tracker for some reason but was always intended to be included, it now is.

The reason the clean argument was added in a stable release was that all the framework had been created it just had not been integrated.

Looking forward to 0.2.0 whenever that is, will be a feature complete system update utility that is command line based. It will be totally cross platform, I intend to achieve this by having two additional namespaces UNIX and WINDOWS, these will contain the OS specific code, like path names and autopackaging things etc. Hopefully to create a windows package from a Linux package should just be a mater of changing

using UNIX; to read using WINDOWS; and just change all occurences of UNIX to WINDOWS in the Main.cs file.

At least, that's my plan anyways.

Blogged with the Flock Browser

Neil Munro: Warp 0.1.5 released.

Mon, 09 Jun 2008 21:44:22 +0000

Warp 0.1.5 is out, logging is implemented as is a full featured security policy many bug fixes and other things have been applied. Work for 0.1.6 has started and will be out as soon as I can finish it. Warp is a massive challenge and the more I develop with Warp the more I learn, which at times it just feels like it's going to split my mind in half. If Warp is useful to even one person then it is worth it. If there is someone who is following the development of Warp and intends on using it when it is done, could you please contact me, just so I know?

So here it is Warp 0.1.5, it is dedicated to the memory of Michael Bryson, I never knew you, but you were her friend, you should have held on, she'd have taken good care of you! I hope that you have found your peace, R.I.P.

Blogged with the Flock Browser

Neil Munro: Two functions to go!

Sat, 07 Jun 2008 00:44:39 +0000

Warp only has two functions to be written in the logger function for 0.1.5. to be released. I had a bottle and some (didn't quite finish the second) of ale, and wrote a function for viewing the log file within Warp, it was one of the simpler function that just required some pretty brainless person to do it, so in a drunken mood writing mind numbingly simple functions was the best thing for me to do.

I am very much looking forward to releasing 0.1.5 since it has been almost three months since 0.1.4 Warp has doubled in size, a few quite frankly critical features have been added.

As usual after 0.1.5 there will be a much more rapid release of bug fixes and stabilization of these new features which will obviously be 0.1.6 then a longer to be released 0.1.7 I apologize for the long delay in new feature releases however being more or less a one man project it's hard to add everything I wish to add in a short time frame.

However as well as having a list of tasks for Warp I have now placed all OUTSTANDING tasks onto the sourceforge tracker system. This means should something go horribly wrong on my computer there exists another list.

In personal news, I built a flat-packed bed in about 10 min (if we take away time spent reading instructions and mission impossible style moving out of the freshly assembled bed since I constructed it around me, but I'm small it was easy enough to get out of) I personally don't see what the problem is about building flat-package furniture in short time periods. I also now have a wah-wah pedal for my guitar, it sounds sweet, especially when daisy chained with my grunge pedal.

Blogged with the Flock Browser

Neil Munro: Open Source, the American dream?

Tue, 03 Jun 2008 23:54:52 +0000

The American Dream the idea (often associated with the Protestant work ethic) held by many in the United States of America that through hard work, courage and determination one could achieve prosperity.

"America touts itself as the land of the free, but the number one freedom that you and I have is the freedom to enter into a subservient role in the workplace. Once you exercise this freedom you've lost all control over what you do, what is produced, and how it is produced. And in the end, the product doesn't belong to you. The only way you can avoid bosses and jobs is if you don't care about making a living. Which leads to the second freedom: the freedom to starve." Tom Morello.

I would like to explore these statements, and how we as a group of people all under the open source name are actually the embodiment of the American dream.

I do agree with Morello, we have the freedom to be free and die or the freedom to basically be a slave to the wage. But I would ask this, where else can anyone do what they want, when they want, with what they need, anywhere else in the world other than the open source movement?

We are known for being a group of people passionate about something, donating our time, money and attention to various things, all for little or no reward, yet we do not get disenchanted with it, we do not get dejected and we do not give up really. We have all sorts of people doing all sorts of jobs, because we are free to do so.

We have, through hard work, courage and determination achieved prosperity, we have an amazing kernel with support for more hardware than any other kernel on the planet, we have other kernels too, let us not forget various UNIX kernels, solaris/BSD et al, we have literally HUNDREDS of fully fledged operating systems, hundreds of Linux ones, dozens of UNIX ones, several which are neither but are still open source. If they had not prospered in one way or another they would not have survived, it's a fact of life. Our evidence speaks for itself, no-one can deny how good our work is. No one can say it's been unsuccessful, and yes, we've heard 'year of the Linux desktop' many times, but every year we get that little bit further along, we have to just keep working hard, we must have the courage and we must remain determined.

We even have projects than allow you to build a free system from the bits YOURSELF, how much more free can you get? That's right, you can CHANGE the building blocks too! We can do that! We are totally and 100% free!

It is a fact we are more or less a communism, but one that actually works, everyone has the same rights, everyone can (if they want to) do anything they want to do. Being a communism is not a bad thing, it is in fact theoretically the best form of government there is, but it does not work well with human nature. But we have pulled it off here, somehow, and yet, we are free, fully free, we have the freedom to proper from hard work, courage and determination.

SCO took us on, and we triumphed, Microsoft claimed patent law suits against us, and were is the evidence? We are strong, and we are powerful and we are here to stay. When Steve Ballmer calls us a communistic group, we can reply 'So what if we are? We achieved the American dream, almost without trying, did you?'.

If we did not have anything that has prospered we would not be here right now, we couldn't be, we'd have silently disappeared over night without any attention. But we remain, our work speaks for itself, and we are and always have been free.

We as a wide open source community have collectively achieved the American Dream.

My thoughts for the day, mayhap they make sense, maybe not.

Blogged with the Flock Browser

Neil Munro: Autopackage in warp.

Tue, 03 Jun 2008 01:20:19 +0000

Since this blog is now included in planet autopackage I thought I would write about how Warp and Autopackage work together. :)

Warp was originally came out of me experimenting with Linux From Scratch but that never worked out, but I was interested to see if I could solve the problem I came up with. Which was obviously the installation of software upgrades through RSS feeds. I have since proved this was possible, and I later found practical uses and use-cases for Warp so continued to develop it.

In order to get anything done quickly, I opted to use autopackage since I didn't have to worry about much else other than setting the file to be executable and running it. Autopackage allowed me to concentrate on code rather than various packages (they eventually came later) but to have something as easy as autopackage was fantastic. I have always been a fan of autopackage and the installer-esque way it works, while having system updates from the distribution is great, my housemate found it a difficult concept at first that all the software you could run was all retrieved from one application. Autopackage and warp are great for people like him who wish to run Linux but want an easy way of installing latest, or non-packaged software.

Once warp is done it would be possible to subscribe to the package database RSS feed on the autopackage website and have updates brought straight to your computer. I intend on making Warp graphical at some point and this will enable him to get the latest pidgin, firefox etc and if I can learn how to use autopackage (fingers crossed) I can just write a new entry to an RSS file and warp will update him. Again if I can autopackage the packages then he doesn't have to worry too much about command line output, and really, neither do I, coz autopackage will deal with dependencies, I will write Warp with the ability to compile from scratch and catch make errors etc, but it will be better from his point of view for autopackage to handle the dependencies or lack there of.

I do honestly wish all packaging solutions were as easy as autopackage (to install that is) the goals of autopackage and warp are similar, neither are designed to be used to maintain something such as glibc or x11, but for end user applications such as pidgin or firefox the solution is fantastic.

Perhaps with work warp could be used as an autopackage update tool, but that is pure speculation on my part, and perhaps a little wishful thinking that something I created might be useful somewhere in the world. ;)

Blogged with the Flock Browser

Neil Munro: The poor butterflies :(

Mon, 02 Jun 2008 01:16:10 +0000

So Project Peach was released on the 30th and I just watched it today, I also made a point of ordering a DVD this time, coz I can't for the life of me find one for Elephants Dream. I watched it, and it was funny in a very dorky way. However the antagonists killing the poor flutter-by's in cold blood was a bit upsetting, I read Lovecraft unaffected where characters either die, go mad or an odd combination of both in either order, but animated butterfly's being killed, upsets me? This year has been quite a rough year for me, two relationships that didn't work out (go on, laugh at me only being 21 and wanting something to actually work) and I got more upset over a rendered butterfly being squished.

Anyways, it's a fantastic movie, even if it is only 10 min long, the fur and 'fluffyness' is very well done, the facial expressions of the rabbit were amazing, I swear I could see the cogs going in his mind sometimes. There's moments you wanna laugh out loud at the things the characters get up to, others you genuinely feel for the character. I'm a greatly emotional person about anything, but to be able to get the audience to feel for the character was the greatest achievement of the project.

Blogged with the Flock Browser

Neil Munro: Banshee

Sat, 31 May 2008 01:35:15 +0000

So I'm following the progress Banshee media player is making, using svn, and I have to say every time I update I can see progress, ever since switching to GNU/Linux about 5 years ago I've been looking for a GOOD media player, and I've tried rhythmbox, Amarok, RealPlayer, Helix, all sorts of players that have come and gone, and when we look at the windows world who's version of RealPlayer has been able to import media libraries (I believe), windows media player and iTunes none of them even came close to the slick multimedia experience that a windows media player has.

Until Banshee, admittedly the 0.X releases were no-where near as good as the 1 release will be, but Banshee has the multimedia experience that I feel has been lacking on the GNU/Linux desktop, I am overjoyed to see such a complete application that integrates well with my life, my devices, my services etc.

The play queue feature especially looks like it will be very good, the ability to play video as well as music is very welcome, in fact I have been waiting for such a feature for a long time, it's part of what I mean be a complete media experience, who ever said media was limited to music?

The sheer speed of 1 over 0 is amazing, my one gripe with Banshee was it could, on occasions be rather slow, but that issue has been addressed and many new features added too! Who could ask for more?

So a big thank you to the guys at Banshee for this wonderful application and keep up the good work, it's appreciated.

Blogged with the Flock Browser

Neil Munro: Warp goes subversion

Thu, 29 May 2008 12:43:08 +0000

Warp has now gotten a svn repo!

You can check out the source like so:

svn co http://warp.svn.sourceforge.net/svnroot/warp

It was my job all day yesterday learning how to use and set up subversion, it was hard at first and then everything just fell into place as to how obvious it was. So after months of hacking away, I now have a dedicated version control system for warp!

Enjoy.

Blogged with the Flock Browser

Neil Munro: What happened if there was a free software band like Rage Against The Machine?

Wed, 28 May 2008 14:29:22 +0000

Just putting it out there, been listening to a lot of Rage Against The Machine lately and it's got me thinking about software freedoms and rights etc. Mostly about how most of my friends don't seem to care, I mean I have gotten most of them to use Firefox and pidgin but only one, after several years of watching me use Ubuntu he made the switch, he duel boots, but he finds that it does everything he needs, except gaming, but that's to be expected at this point.

So with this broad apathetically apathetic attitude how do we, for the most part geeks with a passion that other people largely ignore get heard?

Well the answer is simple, however the execution is not. We simply make software freedom cool. It's as simple as that. Which brings me back to my original idea, if a musical group was to achieve mainstream success and actively supporting software freedom, it could potentially influence and reach many new people. Because the band would be seen as cool, hard edged and political people who heard the music may be interested in the causes and goodness knows it in the free software world there are LOTS of causes.

Admittedly I'd prefer to see a sorta metal/rock band do something like this, but that's just personal preference.

Fighting for freedom on the front lines of the gnu generation
Coming at ya from every continent, country, race and nation
You can't stop this movement, we are legion, we are many
Get involved, take your place, set the pace, it takes any

You know, just an example. :)

That's my random thoughts for today, or at least for now.

Blogged with the Flock Browser

Neil Munro: 12 Step Test Plan For Security Policy.

Tue, 27 May 2008 00:42:17 +0000

1) Print Filters.
2) Create the white list from the create filter option.
3) Create the black list from the create filter option.
4) Create white list when one already exists.
5) Create black list when one already exists.
6) Edit white list by adding a new extension (both with and without presence of a file already).
7) Edit white list by removing an extension from the file (both with and without presence of a file already).
8) Edit black list by adding a new extension (both with and without presence of a file already).
9) Edit black list by removing an extension from the file (both with and without presence of a file already).
10) Delete white list (both when there is a file and when there is not).
11) Delete black list (both when there is a file and when there is not).
12) Check for conflicts.

1) Created two files using nano whitelist.txt and blacklist.txt and populated the files, ran Print Filter and the contents of the text files were displayed correctly.

Backed up the two filters and deleted them.

2) In the filters section of Warp I opted to create a new filter, chose white list. Opted to add elements the same as which were in the original file created for test 1. To double check 1, I ran that again the elements were printed correctly.

3) Followed 2 but chose to create a blacklist, opted to add elements, re-ran 1, the contents of both black and white lists appeared.

4) Same as 2 but did not delete the original files, Warp read in the contents, deleted the file and just dumped the contents is just read back into them.

5) Same as 4, but using black list instead of white list.

Deleted list files.

6) Got to the filters menu, and opted to edit a file, warp should automatically create a file it one does not exist. Selected white list, input the same values I have been using so far. whitelist.txt was created on the fly and the user entered values were stored into it.

7) Starting from the filters menu selected to edit a file and to remove values from it this time, warp presented the contents of whitelist.txt in the form of Filter 1: <extension> and asked for an integer value which represented an extension. I chose to remove one, it was removed.

8) Same as 6, except with a black list, it worked as 6 did but using black list specific code.

9) Same as 7, with black list specific code.

10) In the filters menu, chose to delete a file, selected whitelist.txt warp will see if it can find the file, and if so, delete it, works as expected.

11) Same as 10, just specific to a blacklist.

12) From filters menu selected check for conflicts, if there IS a conflict (I created one by ensuring the sh extension was in both files) you are asked to specify with file you wish to remove it from, I opted to remove it from the blacklist, it was removed and retained in the whitelist.

Blogged with the Flock Browser

Neil Munro: Security Policy

Tue, 27 May 2008 00:14:59 +0000

As of about 10 min ago the security policy code was just finished and appears to work. I will include the 12 basic steps I took to test this, but as far as I can tell, the security policy code is in a working and usable state. The next new feature planned for 0.1.5 will be logging when and update is downloaded it's presence will be stored in an xml file.

For example:

Warp download a deb package for the respective system. In my case, Ubuntu. Warp will run a process to pass the package to dpkg and install it, the package will then be able to be removed via synaptic or via warp, i imagine if you un-install a package installed by warp via synaptic the log file will then be invalid, but in this case, warp could run a basic check to see if the package is still there and if it's version number matches, if not remove the record of that package as it's obviously no longer on the system.

A lot of work will be going into this new class and I imagine if I found the security policy functions hard this will be even more so, but it has to be done. It's also worth pointing out I had pushed some things upwards to 0.1.6 for stabilization etc, and it came about I needed to implement them in 0.1.5 so by accident I already have about 20% of my work for 0.1.6 done already.

I've been working on Warp more or less exclusively for under two weeks and more progress has been made in this week and a bit than the last couple of months put together. I apologize, exams and some personal things have deterred me, I had a muse, then lost her, and found another muse, and then lost her too. Somehow though, I do my best work without a muse. Probably because I don't need to spend time with a muse if I don't have one, time of which could be used to develop Warp. I'm just musing here, you understand and seeing how many times I can fit muse or any word based from muse into one paragraph, yes, security policy code was hard and boring!

Blogged with the Flock Browser

Isak Savo: Autopackage 1.2.5 RC1

Sun, 25 May 2008 17:58:34 +0000

I figured it was time for an autopackage related blog post in planet autopackage so here goes:

After a long time in limbo, a new bugfix release of Autopackage has been made. Last week, I released 1.2.5 RC1 which so far has only had successful results in testing. If no issues pops up, it will be renamed to 1.2.5 and an official release will be made.

But before that, I need help testing it some more. So far, it has successfully been tested on:

Ubuntu 7.10 and 8.04
Fedora 6 and 9

The more people who are testing it, the quicker I can get the release out.

Everything you need to know about the release and how to test it can be found here:
Release Page:
http://trac.autopackage.org/milestone/1.2.5
Mailing list announcement:
http://article.gmane.org/gmane.comp.autopackage.devel/6624

After you've tested it, post a comment on this blog, write me a mail, tell me on IRC or send a mail to the mailing list. Contact information can be found here.

UPDATE: 1.2.5 is now released.

Neil Munro: Pitch Black Progress.

Fri, 23 May 2008 01:41:23 +0000

Warp 0.1.5 is coming along very quickly, but sadly not quick enough for my liking, it's mostly implementing new features that is keeping me from releasing so far. Two major new features were planned for 0.1.5 but essential for any serious use as an update tool, so they need to be written as soon as I can. However as I write things, and realize how much work they require I realize my original design was not sufficient so while I worked out mostly how to implement it, I often need to re-design things towards the end of my solution to make them more efficient.

I can say about 500 lines have been added to Warp since 0.1.4 with a better design I can reduce that I imagine, but that shows you the work I have been putting into it. That increases the size of Warp by almost 1/3 of 0.1.4.

Additionally I have been working on the goals of 0.1.6, perhaps a bit premature, but as I finish things in 0.1.5 I see what needs further stabilizing work, as I maintain odd numbered releases are new features even numbered releases are generally more stable. So something that mostly works and needs perhaps just a little attention is pushed forward into 0.1.6 so I can concentrate on 0.1.5.

The two things really holding back 0.1.5 are logging, the ability to undo changes etc. No work has been done on that as of yet. However I am in the process of implementing white and black lists, this allows the user to enable shell scripts by default, scripts will be OFF by default and the suggested security method will be a white list, but I intend on having warp being able to work with both black and white lists. However, if a white list is present the black list is essentially ignored, since a white list is default deny.

It is this new code which has largely grown quickly out of control, however with it mostly implemented now, I can tidy it up and compact it and make it more efficient, at least I plan on doing that. But that's a job for tomorrow.

Blogged with the Flock Browser

Neil Munro: Current changelog.

Thu, 15 May 2008 23:28:46 +0000

Warp 0.1.5 is still not ready. However work is now able to continue at a much faster pace now exams are officially over.

In the mean time I thought I would post the most recent changes since any real progress was commented on in this blog.

16/05/08

1) Have warp working in quiet mode totally now.

11/05/08

1) Added code to manually select which packages are to be installed.
2) Bug discovered, when 0 packages are marked for installation, warp states there are none to install, and then asks if the user wants to install 0 packages: Fixed.
3) Removed pointless code.
4) Moved the code that allows the user to begin to install the updates into a new function, now only have to tie the functions together. After sleep.
5) Fixed the upgrade arguement so it actually upgrades the software as well as updates the feeds.
6) Discovered a bug similar to 2), will check it out in the morning, reproduce by checking for updates, then running warp with the update arguement.

29/04/08

1) Shell script code not working at all as expected, need to look into this.
2) Bug discovered, if you run an update, and download fails, and try to update again, the update count adds one for every time it loops.
3) Bug subsequently fixed.
4) Better worded a bit of dialog.

28/04/08

1) Commented out the case "cli": from vRunArgs, once a GUI is present it'll go back in as an option, but for now it's
just taking up a bit of space.
2) Tested shell scripts, not working as expected, might be my shell scripts though. Actually was what yahoo added to shell script.

22/04/08

1) Tidied up the vReset function, made it work on ANY system with a .Net implimentation, not just Linux and Windows.

24/03/08

1) Have the menu looping until the user opts to exit, or the program is invoked with arguments.
2) vReset has now got windows and unix specific code, meaning the function is now cross platform.
3) Minor athestic thing, all occurences of ex.Message have been replace with ex.Message + ".".
4) Quiet mode argument means if the user does not want to see lots of output, it's not displayed.
5) iDisplayFeeds( ) has been made more efficient, using the array instead of doing file i/o calls, far more efficient now.
6) For every streamobject.Close( ) there is a corropsonding .Dispose( ).
7) Experimental shell script support is now implemented.
8) Reset all the formatting back to my preference after visual studio imported it and changed my preference.
9) Tidied up vUpdateFeeds() to use the array and not create file i/o objects when there's no need to.

Enjoy!

Blogged with the Flock Browser

Neil Munro: Normal services will resume shortly.

Sun, 11 May 2008 00:22:27 +0000

I must appologize for the lack of updates, Warp has not stopped, it has stalled as I deal with various things in life, Uni, among other things, it's kinda the biggie right now. However this is a software blog not a 'etc etc etc, whiny whiny emo shit' blog.

As I have had time to work on Warp I have, nothing major added really, lots of little things slowly making their way in. As it happens I have just implimented a feature to allow total user control over the management of updates. Only took about an hour. Nothing major, but it's kinda critical to the point of warp. Does that make my statement an oxymoron?

Who cares?

I have a list of features I am working my way through I am about 65% of the way through it, however as I find bugs (not features) they get added to the list. So it may yet get larger as I attempt to make it smaller.

New things have been added, cruft has been removed, the cycle continues. Over summer I plan to be working on two perhaps three projects at once. Two C# projects and one personal project, which will teach me python. I figure at the sheer popularity of the language I should learn it.

Blogged with the Flock Browser

Neil Munro: Sweaty Badger (Pt I: The Beginning)

Thu, 27 Mar 2008 23:55:01 +0000

Mr Steve, is the sweaty badger, he is not real, well, to say he is not real is to say Vampires aren't real, Vampires do exist, mimetically speaking, some people believe they are Vampires and there have been documented cases of Vampire-ism, it's just never been proven someone can have a biologically dead body and posses supernatural abilities. Just as it's not possible to prove a badger may be covered in it's own sweat.

So Mr Steve, the sweaty badger exists as an idea. He was the brain child of me and my house mate Tom, when we were either A) Bored, B) In a random mood C) Have been drinking or D) An unknown combination of all the previous.

Mr Steve is a badger, and he is sweaty, he has a perpetual 'WTF?' sad expression on his face, because he knows he has no sweat glands yet he is covered in sweat, you just wanna pick him up, but when you do, you find the sweat is kinda gross and smelly and the desire to pick him up and cuddle him.... just goes away.

Mr Steve has integrated himself into our daily lives, I will wake up and it appears my tasks for the day are to:

1) Take sweaty badger to house training classes as Chris' bedroom beginning to smell.
2) Touch sweaty badgers, sweaty place....
3) Wash sweaty badger.
4) Figure out exactly why sweaty badger gets sweaty after a bath.
5) Sniff the sweaty badger in it's sweaty place....

Naturally I would do at least some of those things, if only we could find this badger, so with nothing more ado...

If anyone has a stuffed badger or plush badger or are willing to donate any variety of badger to me, I'd appreciate it. PM me if your just crazy enough to do so.

I'm a student.... Enough said, but seriously, badgers would be appreciated!

Also, found a cool mix of Funeral For A Friend and Saves The Day, check it out here: The Legion of Doom - At Your Funeral For a Friend :
http://home.lyse.net/supreme/mp3/mp3/The%20Legion%20of%20Doom%20-%20At%20Your%20Funeral%20For%20a%20Friend.mp3

Blogged with the Flock Browser

Neil Munro: 1100

Sat, 22 Mar 2008 23:33:29 +0000

Warp has reached 1100 lines with some features added and lots of junk removed. iDisplayFeeds was reduced to one for loop, far more efficient now. Its always encouraging when you can look at previous functions and make it work with far fewer lines. I intend to make features as stable as possible, if i can make a new feature stable i will do or if i work out a way to make something stable in an unstable release i will fix it for the unstable release. Anyways just a brief update.

Mike Hearn: mikehearn

Wed, 12 Mar 2008 18:26:44 +0000

I moved my blog to mikehearn.wordpress.com – I got tired of having to maintain it myself. If you notice any problems let me know.

Mike Hearn: mikehearn

Tue, 12 Feb 2008 19:45:06 +0000

Last time, I talked about the basics of Singularity: SIPs, manifests, connections, and software isolation. Let’s have a look at how this applies to driver development.

Drivers are a Big Deal for us, because:

About 85% of all Windows crashes are caused by bad drivers
An analysis of Linux showed that driver code is 7 times more likely to contain bugs than the rest of the kernel
When drivers crash, they take the whole system with them, thus losing the users work and – by extension – users trust.
When drivers contain bugs, they can sometimes be used for privilege escalation. It only takes one privilege escalation for the bad guys to win.

So getting drivers right would be a huge win, both for reliability, and for building a foundation on which we can get security right too.

drivers in singularity

Because Singularity is a type system based micro-kernel, a driver is simply a set of Sing# classes (Sing# is C# with extra bits), shipped as verifiable MSIL bytecode images. It runs as a SIP, like any other program does. Obviously, using C# instantly solves a large class of common bugs – pretty much any exploit with “overflow” in its name is solved by using these languages.

You can’t represent the specialized instructions needed to control hardware in safe MSIL, so we need a workaround. This comes in the form of an unsafe DLL which provides objects abstracting the hardware control instructions. This DLL (Singularity.DriverRuntime) is a part of what Microsoft Research call the “trusted computing base”, but that makes it sound like it’s to do with Palladium and DRM, which it isn’t. It’s easier to think of this DLL just as logically part of the kernel, even though it doesn’t run in the context of the main kernel program. These classes have names like IoPortRange, IoIrqRange and so on, and provide a simple OO interface to programming hardware.

Singularity won’t let you construct objects of these types yourself, you have to use a set of special static method calls. And you can’t invoke them directly either. It seems we’re stuck – how do we get a reference to such an object, if we can’t construct them ourselves? And what’s the point of being so awkward?

The answer is that Singularity requires you to annotate your code with metadata, describing what hardware your driver is for and what resources it needs. This is done in the usual .NET fashion, with attributes that decorate your classes and fields. At install time, this metadata is used to build an XML manifest, describing what resources your code needs, and is input to a compile time transform, which fills out an empty method you provide. This transform is itself a part of the kernel, and is applied when you install the driver – it is what adds the protected call to the hardware abstraction DLL.

In this way, Singularity ensures that it has accurate declarative metadata on what exactly the driver needs. Because you can’t get hold of the relevant objects and thus program the hardware unless you define accurate metadata (and this is enforced by the system), the metadata is guaranteed to be accurate. That, in turn, makes it easy to solve problems like avoiding driver conflicts and figuring out the order in which drivers should be loaded at bootup.

the hole in the plan

This would all be fine and dandy, if it weren’t for The Catch (why is there always a catch?). The catch is this – any driver that can access a DMA capable device can subvert the type system by overwriting arbitrary areas of physical memory.

Direct Memory Access is an optimisation that lets hardware devices directly write data into RAM, bypassing the CPU. It’s very useful and speeds things up a lot, especially when moving lots of data around like with graphics cards, hard disks, network cards and so on. Unfortunately, DMA not only bypasses the CPU but also the MMU, thus a driver that can control a DMA capable piece of hardware can control the contents of memory. In a regular operating system that doesn’t matter, because such drivers are running in kernel mode anyway and are fully trusted. But for Singularity it’s a problem.

Fortunately for us, DRM comes to the rescue! Yes, I’m serious. Newer CPU architectures being developed by Intel and AMD feature something called, appropriately enough, an IOMMU. This does for hardware devices what the regular MMU does for the CPU – regulates memory access by mapping DMA read/writes through a set of page tables. By configuring the IOMMU, you can stop hardware from fiddling with memory. This feature was originally intended to stop people breaking into kernel space by plugging in physical pieces of hardware, which apart from the obvious downsides for DRM manufacturers can also be used to do things like unlock screens. And of course it protects computers from buggy hardware, which is not such an uncommon thing.

It’s also mighty handy for us, because it closes the hole that would mean DMA-aware drivers could break out of their type-verified jail.

how it helps security

Before you can make something secure, you need a threat model. What exactly do we want to stop?

I’m going to skip over that here and mumble something about stopping malware and viruses. It’s vague and not that useful, but it’ll help illustrate these examples.

Back in 2001 the idea of Microsoft restricting what software you could put on your system seemed grotesque. They were just coming to the tail end of the the browser wars, broadband was just beginning its rollout and the general problem of botnets had not yet reared its ugly head. The potential for abuse seemed way too high, the benefits way too low. Despite that, drivers were known to be such huge reliability problems that a signing program was instigated …. Microsoft would provide you with a unit test suite, and if it passed, that suite would sign the driver for you. The test suite checked for many common problems and didn’t require anybody to hand over their code to Microsoft. Seemed like a good idea, right?

Well, of course it was controversial. Worse, it didn’t necessarily work. Rather than pass the tests, some manufacturers cheated. Rather than risk a schedule slip, or pay the $250 verification cost, some moved the mouse and clicked OK on the unsigned driver warning for you. D’oh.

If we’re willing to be a bit despotic and require signing that cannot be disabled for some things, can we do better? Yes, but it would suck to require every driver to be signed. We can avoid this in the most common cases.

For instance, thanks to the declarative metadata, we can now statically inspect a driver and say, well, gee, this driver will only be activated for devices of class “sound card”, and it doesn’t use DMA, and it doesn’t read or write any files (we know this because we can statically verify that it can’t get any connections to the filesystem server). What can this driver do, except mess up the sound card? Not a whole lot, actually. It could still hose the box by misprogramming the hardware, but such bugs are rare. But it can’t blue-screen us, because it’s not running in the actual kernel. If it crashes, the worst that happens is audio is interrupted until the driver restarts and programs re-establish their connections (assuming we write software to handle driver crashes of course – more on that later).

Generally, we can define some set of privileges (in Singularity this is equivalent to defining some set of connections a SIP can access) which might be dangerous, and only require signing for those drivers. What might trigger such a requirement? Accessing system files or configuration is an obvious case. Being a network driver is another case (how to DoS a target without falling foul of the firewall? Become a network driver). Fortunately, i
t’s rare that drivers need access to anything other than the hardware and their own configuration data, and network drivers are typically uninteresting enough that the default drivers suffice for most people. Thus the majority of drivers can be safely written, distributed and installed without Microsoft ever needing to be involved whilst preserving the security of the system.

Isak Savo: Sweet new tool

Mon, 28 Jan 2008 20:10:32 +0000

Most experienced computer users will agree with me when I say that typing something is often quicker and easier than navigating a deep menu structure. I hate the standard windows start menu with a passion, which means I was quite happy when I found the nice open source tool called Launchy. Now just a quick <alt>+<space> away, I can search an indexed db of my launchers. That's on windows...

On Linux, I've been using a combination of terminal (requires you to know the binary name, plus tab completion only matches beginning of strings in bash), panel launchers and the deskbar applet.

Unfortunately, the deskbar applet lacks the sex appeal of Launchy, and it's always crowded with results I seldom want (yeah, I know I can configure it and add/remove plugins and what-not, but it still lacks sex appeal ;-) ).

Via planet gnome the other day, I found a new player in town - Gnome DO! It's a slick little app, very similar to Launchy on Windows, but slightly on stereoids. By default, you just type something and it'll run it for you when you hit enter. But for things which has multiple actions, there's also an "action" section which can be reached by hitting <tab>. So if I type "movie", it will display an icon showing my movie folder with default action to open it in nautilus. But right now, I don't want to do that so I hit <tab> and enter "terminal" giving me the option to open a gnome-terminal in that folder. Slick.

Now if someone could add SnagIt capabilities to the screenshot tool, I'd be an even happier Linux user. :-)

Mike Hearn: mikehearn

Sat, 26 Jan 2008 21:24:18 +0000

Last time I talked about Singularity I rambled about micro-kernels for a while. You probably knew that stuff backwards anyway. Onwards!

**How does it work?**

Singularity pulls off the impressive feat of being a pure microkernel design that nonetheless is ~30% faster than the traditional approach, and about 10% faster than the monolithic approach for a file-io heavy benchmark. How does it do that?

The trick is very simple – they just throw away hardware memory protection entirely. In Singularity, everything runs in kernel mode, and everything runs in the same address space. The MMU, in other words, doesn’t do anything. There are no “processes” in the traditional sense.

Obviously, if that was the _only_ thing they do it wouldn’t be very interesting. 85% of Windows blue-screens are caused by drivers, not kernel bugs (I suspect the rest are caused by hardware failure). Innumerable privilege escalation vulnerabilities can be caused by bad drivers. Every privilege escalation is a gift to the bad guys. Protection against software bugs is what makes micro-kernels useful.

Programs in Singularity _are_ isolated from each other, but the isolation is done entirely in software, using type theory instead of silicon. They can do this, because programs in Singularity are all implemented in a C# derivative called Sing# (although in theory any .NET language could be used, Singularity uses a few features that they added to C#, so other languages would need the same minor extensions).

You already know that in most modern languages like Java and C# you can’t access memory directly. Most obviously, there’s no way to write the following C in Java:

*((char *)0×1234) = ‘X’; // overwrite the byte at location 1234

It’s not just that there’s no syntax for it. It’s that the Java/C# compilers produce a form of assembly language that can be checked by a bytecode verifier to mathematically prove that the program does not do this. Once proven, we can translate the JVM/.NET opcodes into actual code that the CPU can run, confident that if we don’t give the code a reference to an object, it can’t read or write to it. That’s not only good for reliability – we can build a security system on top of that!

**exceptions to the above**

OK, so that was the theory. In practice it wasn’t so easy. Just like theoretically “sandboxed” programs written in C can escape the sandbox by exploiting kernel bugs, the same is true of Java and .NET – by exploiting bugs in the VM or native class libraries (written in C++) the security system can be compromised.

Worse, in some cases, features like reflection can be used to access objects that people did not expect.

Both these problems have caused breaches in JVM applet security in the past. Given how huge the modern JREs are, that isn’t surprising.

**the solution**

Singularities solution is again straightforward – almost all of the system is itself written in C#, with only a small amount written in C++ and assembly language. This immediately makes the core “kernel” robust against the most common types of attacks. Even then, Microsoft are working on research that will let them prove the safety of the tiny remaining pieces of unsafe code.

The building block in Singularity is called a SIP, for “software isolated process”. Singularity retains the idea of a process being something that owns resources, has its own memory space and is independently scheduled, but it’s all enforced with software and mathematics.

A SIP has its own heap, its own garbage collector (you can choose one from several the OS offers, reflecting the fact that no one GC fits all situations), and its own memory pages. So it’s more isolated than some similar approaches (like KaffeOS), in which all code is loaded into a single runtime, and objects can be exchanged between different programs. This has the disadvantage that you can’t easily exchange objects between two SIPs. It has the pragmatic advantage that SIPs can be quite different from each other – not only using different garbage collectors, but also different language runtimes and in-memory layouts. And it means they can be deallocated quickly, without doing a full-heap GC, which could potentially be extremely slow.

Given that we’re no longer using the CPUs hardware modes, it’s no longer clear how we should define what the “kernel” really is. In Singularity, the “kernel” is the software component that connects SIPs together, does memory management, loads new SIPs and handles some other tasks. Logically, it also includes the trusted/unsafe parts of the system written in C++ or assembly. Some of these are actually attached to a SIP, like the garbage collectors, but because they are just trusted to be correct they can be thought of as a part of the kernel.

SIPs call into the kernel simply by doing a regular function call. You need a way to mark the stack so the kernels own garbage collector doesn’t interefere with the SIPs garbage collector, but that’s easy and fast. Thus all the syscall overhead is avoided.

SIPs communicate via “channels”. These are message based pipes, sort of like UNIX sockets, except strongly typed and faster. A channel is actually a mathematical abstraction – sending a message via such a pipe does not involve any copying or fancy hardware tricks. It’s simply updating a few pointers in memory. Because of this, sending even very large amounts of data between SIPs (say between the network driver and a web server) is fast.

**enforcing security**

It might seem odd that invoking the kernel is just a regular function call. Surely that’s not possible? What stops a SIP from simply invoking the instructions to control the hard disk itself?

The answer is that a SIP is shipped (say, on CD) as a set of MSIL bytecode files. These files are not compiled at runtime as in a regular Java or .NET system. Instead, when the software is installed, it is compiled ahead of time into native code after being statically checked using a variety of analyses. Software installation is a privileged operation in Singularity – it’s handled by the kernel itself. Only software installed by the kernel will be allowed to run.

Because you can’t represent CPU specific instructions like “write to this IO port” or “trigger this interrupt” in safe MSIL, the only way for a piece of code to do that is to be linked with some trusted native library that will do it for you. Because the kernel is in charge of software installation, it can verify that only certain software is linked with such libraries, and even then, only in certain ways. Thus, the kernel can control access to hardware resources without relying on hardware checks – by controlling who has access to the CPU (and how) at install time.

**manifests**

In a regular OS, a process more or less corresponds to a program. A few programs have multiple processes, for instance, iTunes installs a program that navel gazes until you plug in an iPod. But generally speaking, one program uses up one process, and actually it’s quite common for one process to contain more than one program – for instance, a web browser that hosts plugins.

Because SIPs are so cheap, it’s reasonable – encouraged, even – to split a single program into several co-operating SIPs. Thus we have a problem – how do we start a program? What even _is_ a program in such a setup?

The answer is that a program in Singularity is defined by a manifest. A manifest is (like in .NET) an XML file describing the SIPs that make up a program, and the connections between them. A manifest is mostly auto-generated from metadata annotations in the code. When you start a program, you actually invoke a manifest.

**drivers in singularity**

This has interesting implications for how drivers are managed. Unfortunately, I blew my word count again. It’ll have to wait for next time.

Isak Savo: Autopackage Interview

Fri, 18 Jan 2008 15:45:01 +0000

At the end of last year, we (the Autopackage developers) were interviewed for an Indian PC magazine. I think I mentioned this in a previous post some time ago.

Anyhow, now the interview has also been published on Linux.com which is cool. In addition, I've put the original interview from the Indian PC magazine online on autopackage.org in PDF form.

Thanks to Samartha who made the interview. Let's hope something good comes out of this (like a horde of eager developers wanting to do nothing else than hack on the next generation installation framework ;-)

Rob Staudinger (AbiWord Packager): 11 Jan 2008

Fri, 11 Jan 2008 20:56:31 +0000

Greatest common denominator

Kind of as a warmup for an exam in theoretical computer science, I have implemented the Euclidean Algorithm in C

 
#include <stdio.h>
#include <stdlib.h>
int main (int argc, char **argv)
{
	int x = atoi (argv[1]);
	int y = atoi (argv[2]);

 	while (x > y ? (x %= y) : (y %= x));

 	printf ("%d\n", x ? x : y);

 	return 0;
}

and GNU Make

 
mod = $(shell expr $(1) % $(2))
is_zero = $(shell if test $(1) -eq 0; then echo "1"; fi)
gcd = $(if $(call is_zero,$(2)), \
      $(1),$(call gcd,$(2),$(call mod,$(1),$(2))))

 all:
        echo $(call gcd,$(A),$(B))

but now I really gotta turn my attention to the more involved assignments.

Mike Hearn: mikehearn

Sun, 06 Jan 2008 15:02:54 +0000

I want to write some stuff about Microsoft Singularity. It’s cool and everybody with an interest in computing should be talking about it. Here’s a summary for those who don’t want to read all the papers.

**what is it**

Singularity is an operating system research project. It’s a team of smart people who were told “what might an operating system look like, if it was designed from the ground up for dependability”.

People on pop forums like Slashdot and OSNews have been wishing for _years_ that Microsoft would throw away Windows and start from scratch, to address problems like reliability and malware. Usually their wish revolves around rebasing Windows onto some form of UNIX, but that’s a crap idea and wouldn’t actually achieve their wish at all. If you want to address problems that are caused by fundamental design decisions, you need to revisit them. This is what Singularity does.

Dependability is a pretty broad topic. At minimum it means not crashing, and it means being secure. But, although the Singularity researchers are exploring many topics, they don’t have a wide-open mandate … it’s not chartered to do GUI research for instance.

**how is it different?**

That’s what I want to talk about here.

Singularity is a high performance, single address space microkernel design, which uses static type verification to enforce reliability properties and flexible pattern based ACLs.

Oooh, there’s a lot of scary academic talk in that sentence. Let’s figure out what it means. This is going to be complicated, because Singularity is pretty different to textbook OS designs.

It’s _high performance_. Performance isn’t actually a goal of the project, but the researchers are smart enough to realise that if they don’t keep it in mind, their research will wander into the weeds and become completely uncommercialisable. There are hints that Microsoft are thinking of one day using this research in real products, so it’s important to be fast (more on that later).

It’s also a _microkernel_.

For now let’s just focus on the fact that it’s a microkernel. We can cover the other things in future blog posts. Skip to the bottom if you already know this stuff – I’ll assume here that you, my dear reader, aren’t entirely sure what a microkernel is or why they are supposed to be slow.

**a microwhat?**

Historically, there are two ways to design an operating system, which are actually the same way but using different cost:benefit analyses for certain decisions. These are whether to use a microkernel, or a monolithic kernel. Note that we’re talking low level designs here … this stuff is all independent of whether you use a task bar or a dock in your UI.

Recall that in a microkernel design, subsystems like the filesystem and network drivers run as more-or-less regular programs outside the kernel itself (which is distinguished from other programs bascially by running in a special CPU mode). The kernel proper only handles starting processes/threads, sending messages between them and a small number of misc things like CPU scheduling. True microkernels are hard to find these days. You’ve probably used them without realising it – for instance, QNX is an operating system designed for embedded applications like Cisco routers, and QNX is a pure microkernel design.

Sidenote: Here’s a quick recap of virtual memory. When your code reads some value from memory, the CPU internally converts that address from a virtual address into a physical address it can give to the memory controllers. On a 32 bit CPU they’re both 32 bit pointers, and you’ll probably never see the raw physical address unless you’re actually a kernel developer. The conversion is done by a component of the CPU called the MMU (memory mapper unit), and is subject to an access control check. Memory is split into “pages”, which are 4kb each on Intel/AMD chips in the standard case, and each page can be mapped independently. Each page mapping has permission bits – read/write/execute – like a UNIX file would.

This memory mapping is the foundation of all security in existing operating systems. It prevents a buggy program splatting another program accidentally, and because only the kernel can update the page tables, and all hardware access has to go via the kernel, it means a program running in user-mode can’t really do anything interesting unless the kernel allows it. And because the MMU won’t let you read kernel memory, you can’t force it to give you that permission. It also means that we can use swap files to let the disk pretend its a RAM chip – just unmap the part of the processes address space that was swapped out, catch the error when the program tries to read from it and load it back in.

Virtual memory is jolly good and is one of the biggest improvements to computer reliability in the past 13 years. Windows 3.1 didn’t use it, Windows 95 did and that right there was why many people upgraded. The advantages of the microkernel then are obvious …. more use of virtual memory means buggy kernel components can’t blue-screen the computer like they can today. If your filesystem crashes, just restart it!

In a monolithic design, filesystems, drivers and even web servers are all loaded into the kernel itself and all run in privileged code. The kernel still provides message passing systems for user-mode processes to communicate, but they aren’t used anywhere near as much. Every mainstream server or desktop OS is monolithic – Windows, Linux and MacOS. Note that whilst Linux has always been monolithic, Windows NT started out as a microkernel, and MacOS X – being based on Mach – is theoretically one today. I don’t know anybody who believes that though.

It can be hard to say whether a particular system is truly a microkernel or a monolithic design, because it’s not a boolean yes/no thing – for instance, Linux runs its graphics subsystem in a separate process (the X server) whereas Windows _used_ to do that but doesn’t do it anymore. Nonetheless, everybody agrees that Linux is not a microkernel. A good smell test is whether the filesystems are running in kernel mode or not – graphics can be a grey area, but the filesystems are generally not.

Anyway, Singularity being a microkernel might seem strange, because historically the debate has in academia always been won by microkernels and in the market has always been won by monolithic kernels, largely for performance reasons. These arguments were going strong in the 80s and you can read the infamous Torvalds vs Tanenbaum debate on it here. So at first it might appear that Singularity is just another academic exploration of the theoretically clean thing to do, at the cost of real world usability. But it’s not so.

He won the debate

**why are microkernels slower than monolithic kernels?**

Microkernels are typically slower than monolithic kernels because there is a cost associated with transitioning between user mode, kernel mode, and back again. What’s more, there’s an additional cost for switching the CPU between two user mode processes: a context switch.

These costs are small but real, and when you do bazillions of them per second can come to completely dominate the CPU such that you’re not getting any actual work done. Measuring those costs is hard, although the Singularity team have managed it.

The reason they cost precious time is because the CPU has to do unusual work to make them happen, and because the majority of the CPUs time is spent _not_ doing that unusual work, it tends to not be well optimized (this has changed in recent generations of x86 chips, but the general point holds).

For example, when invoke a syscall to make the kernel do something, you use a special CPU instruction. That used to be “int $80″ on Linux but these days you can use the “sysenter” opcode on kernels and x86 CPUs that support it (nearly all do). Control then transitions to the kernel. This is pretty fast on modern computers, but it wasn’t always so – and in fact early versions of Windows actually abused an illegal instruction because they found triggering a CPU exception was a faster way to get into kernel mode than using an interrupt (the official way). Intel fixed that

Context switching is more expensive, firstly because it obviously involves a transition to kernel space, so you pay for the cost of doing that, but mostly because reconfiguring the page tables is slow.

Reconfiguring the page tables is slow partly because, again, it’s an unusual operation (it involves poking special registers on x86 chips), but mostly because it requires flushing the _translation lookaside buffers_. These buffers cache the result of the MMUs lookup. Even though MMUs are custom designed hardware and very fast, they’re still not free and yet the translations are needed every time code accesses memory, which is all the time, thus caching makes a lot of sense.

This also makes it hard to quantify exactly what a context switch costs you. We know it costs _something_ because of CPU design fundamentals, but the actual cost is spread out over the code in the new process to run. Immediately after a context switch then, your computer is running a little bit slower, and then picks up steam as the TLB fills up.

So we have two conflicting priorities here. On one hand, using virtual memory to separate address spaces can improve reliability by insulating programs from each other, which is good, but on the other hand, it costs us some hard-to-measure amount of performance, which is bad. Worse, although it’s true that CPUs have got faster over time, they got faster at running code and not at doing address space manipulations, so we can’t rely on Moores Law to bail us out this time.

message passing

Micro-kernels are based on the idea of sending messages between processes running in separate address spaces. Thus to read a file, first you have to send a message from your program to the filesystem server. This means formatting the message in your own memory space (fast), invoking the “send message” syscall (not quite as fast), the kernel then copies the message into its own address space (sorta slow), does a context switch to the filesystem server (slow), and then copies the message into the filesystem server memory space before leaving kernel mode.

Once the filesystem reads the data, you have to do the whole thing in reverse but this time copying the data back in a message …. because the cost of a message send goes up with its size, this is even slower than the initial request!

In contrast, in a monolithic design, you format your request (fast), do a read syscall (not quite as fast), wait whilst the filesystem gets your data, the kernel then copies your data into your memory space (or perhaps the hardware will do that if you’re using DMA) and returns to user mode …. wow, simpler and faster! The disadvantage is that if the filesystem code is buggy, you blue-screen and lose everything.

About 80% of Windows crashes are caused by crappy drivers. So, if you could stop drivers from exploding the system in the same way we can stop apps, we can eliminate 80% of the worlds blue screens! That’s pretty cool. It also means that drivers could have security enforced. Today if you install a 3rd party filesystem, who knows what you’re getting? Unless you review and compile the code yourself, you just have to trust whoever gave it to you. Even if they mean well, a bug in the new driver can open a local root exploit, compromising the entire security system

It’s probably not surprising then that academia preferred the slow-but-robust solution, and desktop OS vendors preferred the fast-but-unstable solution. It’s not that they didn’t try! Windows NT started out as a pure microkernel solution, but even with super-optimized IPC they eventually gave up and moved the whole shebang into kernel space, including the GUI, which they got a lot of stick for but made Windows feel snappier and more responsive thus making users happy.

the singularity approach

Singularity manages to have its cake, and eat it. It gets both the robustness benefits of a microkernel, and manages to get even better performance than a monolithic kernel. Neat!

But I wrote too much above. How it pulls off this trick will have to wait for next time.

cpu photo by nadya peek

Mike Hearn: mikehearn

Sat, 05 Jan 2008 16:48:22 +0000

Here’s a note for the interwebs. The GigaSet SE 551 adsl/cable router firmware is really quite nice, and gets a lot of things right. Sadly, the same cannot be said for the people who run their website.

Unfortunately, like all software, you may need to upgrade it from time to time. The firmware is shipped on their site as a Windows EXE, despite consisting only of a readme file and the BIN that you upload into the router. What is a humble Linux or Mac user to do?

Fortunately, the Windows EXE is only a Winzip self extractor. Here’s the magic incantation to get your grubby paws on it using Linux:

unzip Gigaset_SE551_WLAN_DSL_Cable_V2706_int.exe

Probably, a similar trick will work on MacOS.

Mike Hearn: mikehearn

Thu, 03 Jan 2008 17:41:03 +0000

Looks like Saudi Arabia stopped another planned al-Qaeda attack. From the article:

The conservative Muslim kingdom also said it arrested 208 suspected Al-Qaeda militants over the past few months who were plotting assassinations and an attack on a logistical oil facility.

208 seems absurdly high, and given the KSAs rather dubious standards of justice, it’s unlikely there were actually 208 actual terrorists involved.

Nonetheless, the risk remains of a suicide attack on key pieces of infrastructure such as the Abqaiq stabilisation facility, or the refinery at Ras Tanura. I last wrote about this in April – to recap, approximately 6-7% of the worlds oil supply passes through Abqaiq, and any attack that managed to shut it down would cause an instant oil shock in the west.

That makes it a very juicy target for anybody who doesn’t like us. Back then, only a fairly ragtag bunch of terrorists were trying to damage it. Now we have another risk – if Iran is attacked it will fire missiles not at our ships (which they can easily destroy) but at Abqaiq or a part of the KSA pipeline network. A clever strategic strike for sure, because it doesn’t hurt soldiers …. it hurts the average US citizen, in the wallet.

The Saudi regime takes this threat seriously, and is building up a dedicated army to protect the petroleum infrastructure:

Saudi security officials say the force now has about 9,000 of its planned 35,000 troops. Still to come are a helicopter force crucial for pipeline defense and an air-defense system designed to thwart both suicide aircraft and missiles.

In other news, oil brings in the new year by passing $100/barrel. Cheers!

Mike Hearn: The fuel of 2100: Petroleum

Sat, 29 Dec 2007 19:41:13 +0000

The oil is running out. I talked about this on my blog before. Oil has thousands of uses, but one dominates all the others - petroleum manufacture for our vehicles. Because of this, it’s commonly assumed that leaving behind oil means leaving behind petroleum too … and perhaps even cars. But does it?

New technology currently in the R&D labs suggests another idea - manufacturing our own petroleum in a speeded up, industrial scale version of the same process that nature uses.

Question is, how?

German researchers are working on replicating photosynthesis, with the goal of converting atmospheric CO2 into CO. CO and hydrogen are the main components of syngas which can be used to manufacture gasoline. They’ve managed it too, but their current reaction uses energy from benzene and has low yield. They need to replace that energy input with something else and increase the efficiency for it to be a viable building block for manufacturing gasoline.

Another approach is being pushed by VC firms in the Valley: LS9 is a company genetically engineering microbes to create petrol from biomass. It’s a bit like ethanol manufacture, except that ethanol isn’t backwards compatible with our existing infrastructure (pipelines and engines all need modifications). LS9 claim they have been able to build microbes that eat sugar and expel gasoline … a remarkable feat if true. But like all biofuels, it’ll be constrained by the size of the biomass input. Their website is pretty vague about what sort of inputs the process can take. Even if it can eat more or less anything, there’ll still be huge issues with scaling up such an approach - just moving the biomass to the refineries is as yet an open question, viability wise. They think they’ll be competitive by 2010.

Why it matters

The fact that both techniques can (in theory) produce petroleum is a huge deal. Other than the fact that using it shifts huge quantities of carbon from the ground into the air, petroleum is a really great fuel. It’s amazingly energy dense, easy to transport, and we have a lot of infrastructure built to handle it.

Of both, the German approach appeals to me more. Photosynthesis isn’t particularly efficient and the LS9 approach still relies on it. Although making petroleum instead of ethanol is a big leap forward, as it solves the sticky backwards compatibility issue, the other problems with biofuels (land use, etc) are still open.

Sucking CO2 straight from the air using mechanized, optimized and industrial-scale processes seems like a better way forward. Whilst the air doesn’t contain a whole lot of CO2 proportionally, and using plants lets you cover large areas of land with ease, the ability to build sun-powered carbon-neutral gasoline factories is a very tempting prospect. I need to run the numbers some time on how much air you’d need to process to create a gallon of gasoline, to see how feasible such a scheme might be.

Rob Staudinger (AbiWord Packager): 27 Dec 2007

Thu, 27 Dec 2007 17:04:08 +0000

Not as advertised

Alvaro, the issue I have with this trick is that – while it is fascinating at first – things tend to break once you start to do slightly more than the most simple coroutine body with sequential control flow. People like to forget that Duff's Device was conceived as a manual loop unrolling technique and abuse it for coroutines, praising the fact that it is all plain C. I would argue that the handful assembler statements for a proper coroutine implementation are well worth the effort and result in a cleaner, less fragile implementation. Almost inevitably, when the programs grow more complex, at one point you'd love to have a loop counter or some other local variable. This is when the plain C restriction leads to nasty workarounds. Static local variables just don't scale very well.

The article itself brings it to the point:

[...] The [PuTTY] SSH protocol code contains real-life use of this coroutine trick. As far as I know, this is the worst piece of C hackery ever seen in serious production code.

Aaron Spike (Inkscape Packager): pyToddler 0.3

Mon, 24 Dec 2007 00:39:00 +0000

I would like to announce the release of pyToddler 0.3. Huge thanks to Michael Greb for the inspiration and some code for the new activity selection menu. Rush out to the Official pyToddler Website and grab yourself a copy.

Isak Savo: Quote of the week

Mon, 10 Dec 2007 15:59:15 +0000

The salesclerk in a shop in Bangalore, India, when finding out I am from Sweden:

Clerk: Oh, we once had someone famous from Sweden in this shop. Maybe you've heard of him, his name was car.. carl? carl-gustav!
Me: "Carl Gustav"? Yeah I know of him. He's the bloody king of Sweden.

Aaron Spike (Inkscape Packager): CC Licensed Guitar Plans

Fri, 30 Nov 2007 03:04:00 +0000

Today I stumbled upon a real gem while searching for information on building acoustic bass guitars. Liutaio Mottola provides a wealth of information about lutherie. I'm most excited about the collection of Creative Commons licensed guitar plans. I hope many more artisans will follow in sharing their detailed designs without cost and under a license that allows the whole online lutherie community to be enriched. Huge thanks to Liutaio Mottola.

Aaron Spike (Inkscape Packager): Little Gem

Fri, 30 Nov 2007 03:02:00 +0000

The Little Gem Amplifier is plastered all over the internet. I was beginning to feel like I was the only guitarist on earth who hadn't built one. No longer! Here's the proof:

Isak Savo: Web 2.0 continue spreading like a plague

Wed, 28 Nov 2007 09:57:18 +0000

I'm both amazed by and a bit reluctant about the move to web applications. I recently praised Google Docs for the possibility to collaborate with other people, and I'm a happy user of other nice web apps: GMail, Google Reader and very recently started to play FallenSword, which is a semi addictive online MMORG played in the browser (html+javascript, nothing fancy), In short, Web based applications are great, where they make sense.

But when I needed to install VMWare server the other day, to be able to build linux packages on older systems, it was with a mixed feeling of technical amazement and complete discust. Remember VMWare 1.x, where they had this user friendly application to manage and run virtual machines? It's now gone in the 2.0 beta. All that is left is a butt ugly, slow and counter productive web implementation. You even run see the guest os in a tab inside firefox - ugh.
Granted, the product is only in beta right now, and will surely see some improvements in both speed and "prettiness", but I can't really see the benefits of this move.

And it's not even a "real" web application. It's a locally hosted program that installs itself as a web server listening on port 80. This means the nice benefits of web apps (universally available, transparent upgrades) are limted or gone, while all the negative sides (slow UI, poor integration with the desktop, inconsistent interface etc) are still there.

I'm glad I was asked today to fill in a survey on my experiences with the beta version, and I think I'm not the only one with negative comments about the new interface. Hopefully, they have a backup plan containing the old, usable standalone application.

Planet Autopackage

Jan Niklas Hasse: Google Mail's Anti-Spam Filter becoming stupid?

Jan Niklas Hasse: LinuxTag 24-27 June 2009

Isak Savo: Perl was born

Isak Savo: Disconnected from reality?

Jan Niklas Hasse: Autopackage @ LinuxTag in Berlin

Isak Savo: First shot at synchronizing with the iPhone

Isak Savo: Stuff they don't tell you about the iPhone

Mike Hearn: mikehearn

Jan Niklas Hasse: Autopackage 1.4.0

Isak Savo: Living with multiple operating systems

Isak Savo: úâ¶¯ Þ3!Ú¦Ø§ú0z·o"Ú²×æ

Jan Niklas Hasse: Autopackage 1.2.6

Jan Niklas Hasse: Autopackage 1.4 progress

Mike Hearn: mikehearn

Isak Savo: Back from PDC

Isak Savo: Attending PDC08 in LA

Isak Savo: Do you really need touch screen?

Isak Savo: Humanity

Neil Munro: The road to 0.1.6

Isak Savo: New programming site

Neil Munro: Title

Jan Niklas Hasse: GtkMencoder's Trac page and hybrid autopackages

Neil Munro: Progress on 0.1.6.

Neil Munro: Update

Mike Hearn: mikehearn

Mike Hearn: mikehearn

Neil Munro: Protection charms.

Neil Munro: 0.1.6 progress

Neil Munro: Warp 0.1.5 released.

Neil Munro: Two functions to go!

Neil Munro: Open Source, the American dream?

Neil Munro: Autopackage in warp.

Neil Munro: The poor butterflies :(

Neil Munro: Banshee

Neil Munro: Warp goes subversion

Neil Munro: What happened if there was a free software band like Rage Against The Machine?

Neil Munro: 12 Step Test Plan For Security Policy.

Neil Munro: Security Policy

Isak Savo: Autopackage 1.2.5 RC1

Neil Munro: Pitch Black Progress.

Neil Munro: Current changelog.

Neil Munro: Normal services will resume shortly.

Neil Munro: Sweaty Badger (Pt I: The Beginning)

Neil Munro: 1100

Mike Hearn: mikehearn

Mike Hearn: mikehearn

Isak Savo: Sweet new tool

Mike Hearn: mikehearn

Isak Savo: Autopackage Interview

Rob Staudinger (AbiWord Packager): 11 Jan 2008

Mike Hearn: mikehearn

Mike Hearn: mikehearn

Mike Hearn: mikehearn

Mike Hearn: The fuel of 2100: Petroleum

Rob Staudinger (AbiWord Packager): 27 Dec 2007

Aaron Spike (Inkscape Packager): pyToddler 0.3

Isak Savo: Quote of the week

Aaron Spike (Inkscape Packager): CC Licensed Guitar Plans

Aaron Spike (Inkscape Packager): Little Gem

Isak Savo: Web 2.0 continue spreading like a plague

Isak Savo: úâ¶¯ Þ3!Ú¦Ø§ú0z·o"Ú²×æ