Interview with the Core Developers

Obsethryl's Lab has posted an interview with Bruce, Chriss, Sander and I. It's a treasure trove of info about SharpOS, check it out!

Busy Bees at the Circus

The project has been a hive of activity the past several weeks. Our first release at the beginning of the year has spawned a constant stream of chat from interested parties, and all sorts of code contributions. Within a week of our release, someone hacked together a Snake game to embed in the kernel, for example:

(A screen-shot of Snake# running on our kernel – making it rather easy to get distracted from debugging other kernel issues.)

With all the people stopping by, it's been a mad house. People have submitted patches for bugs we didn't know we had. And we've even had a bunch of contributions for command-line functionality – even though the command-line was only meant for demonstrative purposes!

One interesting sample of work on SharpOS has been an attempt at VGA graphics…

(Somebody took one of our logo submissions a bit too far…)

…but don't get too excited – the patch hasn't been trunked because most of our hardware stuff is pending until after our runtime work.

Another screenshot which reflects some miscellany that has been added to the kernel:

(Don't worry; I think that area of memory is supposed to look like that.)

…that's right, a memory dumper! Oh well, it's exciting in a nerdy way. And notice at the bottom, that William has been working on a software clock.

But the most important change in the code-base over the last couple of weeks is one that a screenshot really can't do any justice for. Chriss has added support for object instantiation - Real, genuine, live objects. And as a result, the most important and exciting work has been centered on what this means.

William has been working diligently to get type metadata into the kernel, and for the most, part has succeeded. And as a side note – it's caused some big changes for us. The kernel no longer fits on a floppy image! So because of a bug in QEmu for Windows, which is the only emulator that still supports our over-sized disk image (other than the more-buggy Bochs) – we are now including the binary for QEmu for Windows, in our repository. Our NAnt build files have been expanded – so give 'nant run' a try – and an identical functionality for Visual Studio is still being worked on.

Sander has also been working on kernel design and driver implementation. He did a much-needed cleaning of the wiki (you can actually find your way around in it now!) So big kudos to him for that, because we now have significantly less redundant questions about project information that should be retrievable from the website. And due to his work with driver implementation, at the very least, he has stirred up much-needed debate on what needs to be done, so that we can move forward as a group.

(We have a contributor, named Sam Wilson ("tecywiz121"), who has been writing a C# implementation of ISO9660, so that we will both be able to use other PC emulators again, as well as the same sort of functionality inside the kernel, when that time comes.) He says it is coming along well, so you can imagine that has everyone excited.)

But type metadata means a lot. It means we are months or even only weeks away from having reflection. And after we toss a Garbage Collector in, we'll be able to start implementing the stubs to support the Mono mscorlib – which means we are probably only 4 or 5 months away from a JIT. (Because remember, our AOT just needs some slight refactoring so that it can be AOTed to sit on top of an also-AOTed Mono mscorlib and Mono.Cecil).

 

We also have a contributor, David Stetz ("codan3"), working on threading (and mostly done with the base implementation.) And Chriss is working on exceptions! So even though only a month ago, our code looked dirty and dead-ended with pointers and frivolous exercises at re-inventing the standard C library, we are jumping in leaps and bounds. We are now starting to have reasonable conversations about object-oriented hardware abstractions, and the like.

Oh, and Sean Donnellan ("Donno") has updated our "SharpBot" (in IRC as "SharpOS"), to actually be rather functional, so we now have constant logs of everything said in the chat room. (Don't worry, the topic has a disclaimer.)

 

And a final screenshot for this post, demonstrating a listing of PCI devices:

(Listing device information is important, even if you can't understand it.)

The PCI work was done by Cédric Rousseau and Stanisław Pitucha ("viraptor"). It will undoubtedly be ported to OO and expanded, very, very soon.

Everything is all very exciting, and nothing seems to be slowing us down!

SharpOS Release

We've gone and finished our proof of concept release! And we posted the news on OSnews. The release downloads are on Sourceforge.net.

It's actually been a couple days since the release hit the Web and we've got some impressing stats already. Our sourceforge download stats are around 450 downloads in the first two days, which we're excited about. A number of cool people have come to visit us in IRC as well, and maybe some of them will want to help us out with our coming milestones over this new year. So far (except for in IRC), we've got only a few comments, but the news is quickly spreading to other websites. To top it off, I managed to not mess up the release too badly (something that is difficult not to do for first releases).

So, if you haven't already, go and get SharpOS from our downloads area and try it out!

We Are Busy

OK, so Bruce has done an amazing amount of work on the SharpOS kernel's demo shell, which provides an interface for users to inspect and play with the SharpOS kernel as it currently stands. It is still rather raw, but the remaining bugs are being fixed and once that is done we can begin looking at the first milestone release of SharpOS. Today we've been busy organizing our bug tracker, closing old bugs and opening new ones, assigning them to the milestones etc. I wrote a draft article for introducing SharpOS to the community, we came up with an idea of using a chicken and a cracked egg shell for our mascot, we've got a few new faces around the list and IRC channel, and the usual work on mscorlib stubs continues. The IRC channel has been ablaze with discussion, some important, some not, and our list has been equally busy.

The bottom line is, after a year and two months we are almost ready to show our proof of concept to the world! I don't know about the rest of us but I am at least mega-excited.

PS: I did nothing on Nash today so far.

Pointers, and Structures, and Strings... Oh My...

Ye machine language is an unforgiving beast, aye, she is...



Its true folks, the demo-shell is finally getting off the ground. As you can see, I've added a couple commands. There are still a couple of very small bugs - but the majority have been hammered out.

Some things to remember when you are coding an OS:

1) Your code is what you code. An infinite loop is always an infinite loop. Even if you don't know you coded one, and can't figure out why your kernel "locks up" - and then revert all changes and try again - if you code an infinite loop twice, then you code an infinite loop twice.

2) Pointers are not toys. They are pointy. If you play with some (instead of others), (like writing allocation node consolidation code to prevent memory fragmentation), the ones you aren't paying attention to, will twist together and stab you from behind. You cannot have tunnel vision when playing with pointers, they are unforgiving.

But basically, I've journeyed along the the managed-brick road, and come out with a demo-shell that people can start playing with. In fact, if you download SharpOS from our Subversion repository's trunk right now, and compile and boot it, you will see the demo-shell in action.

There is one bug I'm still trying to figure out. The first command or two, in the linked list of commands, always seems a bit corrupt. Sometimes the first command in the list is completely swallowed into the void. And other times, the list will have two seemingly corrupt commands off the top, with the complete list of expected commands tacked on the end. I've double checked all of my initialization code, and linked-list management code, and I haven't figured it out yet.

More commands, (and obviously a bug-fix or two), are still on the way...

SharpOS Kernel + UNIX

For those of you who don't know me, my name is William Lahti. I am one of the original developers of the SharpOS project, and have contributed various odd parts of the kernel, mostly amounting to organization. At some inspired point I also wrote the foundations of a modern, network-transparent C# window system dubbed SharpWS.

Due to my inexperience with kernels and rather shoddy performance at programming one, I decided that I could help the project without doing too much design work on my own if I were to create a UNIX foundation (libraries and commands/utilities) that could be used natively on SharpOS (written in C#).

So, I started with the shell. I started the Nash shell, which by now has actually gotten quite workable, except for a few bugs in mono that are making things difficult. Currently, Nash supports variables (dynamically and statically typed for future interoperability with scripts), aliases, complete line editing and autocomplete, a set of builtin commands, support for interweaving script calls with Nash commands, the foundations of pipeline support, correct handling of most common special variables/environment variables, and more. It strives to be at least POSIX-compatible, if not BASH-compatible. Oh, and it runs programs alright too. It's still got it's issues, though. Currently, Console.Cursor[Top/Left] are not updated accordingly by Mono if other programs write to the terminal, but a workaround is in progress.

Now, what's the fun of having a managed UNIX shell if we don't have managed UNIX utilities? Sure, I could use Nash with my local GNU utilities, but that won't be the case when SharpOS is ready to run this bad boy. So, I started implementing UNIX utilities. So far I have started two packages of UNIX utilities. 'coreutils', like GNU coreutils, contains the most essential tools... I've implemented cat, cp, cut, date, dirname, hostname, kill, killall, ldd, less, ls, mkdir, pwd, rm, sleep, touch, tr, uname, and which.

The second is netutils, which so far only has one program, 'ping'. Unfortunately, Mono decided not to implement the System.Net.NetworkInformation.Ping class, because it would require super user powers. I disagree, but for now I can't test ping :-\.

I've started three libraries: SharpOS.Unix, SharpOS.Unix.Tar, and SharpOS.Unix.Curses. SharpOS.Unix implements common code for handling UNIX permissions, command-line options rules, timestamps, etc. SharpOS.Unix.Tar is a managed interface for reading and writing tar files. Yes, I am aware that SharpZipLib supports Tar, but I only discovered this fact after I finished the first commit of SharpOS.Unix.Tar, and SharpZipLib is not as powerful or accessible as this API is. Finally, SharpOS.Unix.Curses will be a managed terminal interaction API inspired by ncurses, and using terminfo terminal descriptions from Linux. Right now, only the terminfo code is completed, and the terminal code is not yet committed.

That just leaves ntar, which is a (quite-functional and compatible) implementation of tar, using the SharpOS.Unix.Tar library.

I've tentatively been referring to the whole set of tools as 'nunix' and/or SharpOS.Unix (the namespace of libraries and program classes).

Okay, this blog post got really long but I talked about all the new stuff I'm doing, so there! I'll try not to make my next ones so long :)

Getting the Juices Flowing

Well folks, if you've made it here, you can see that SharpOS now has a development blog. On the other side of things, if you have no idea what SharpOS is, then I should point you at our website – http://sharpos.sourceforge.net/ However, the website is under construction as part of a move (back) to http://www.sharpos.org/ . So, I'll introduce you to the project.

The SharpOS project aims to create an operating system using 100% C# code. A lot of programmers will respond to that thinking, "How is that even possible?" C# is a managed, JITed language. It compiles to CIL, which creates a safety net by being completely compiled into native code, on the spot by the .NET (or Mono) runtime, only when the program gets executed. Unlike C/++ or Assembler, it has no constructs that make it immediately useful on a low level. But, it does have pointers. And that's all we really need.

The founding members of SharpOS decided to tackle the conundrum at hand by writing an Ahead-Of-Time compiler, (in 100% C#), that takes our kernel .NET assembly, and generate a bootable kernel image from it. That is, it compiles the CIL of the kernel DLL into native code. To get around not being able to do low-level machine language stuff, the AOT contains a class with static method stubs on it, (which in and of themselves do nothing). There is a stub for every x86 assembler instruction. And when the AOT compiles the kernel, and encounters calls to those method stubs, it converts them into the correct native instructions. This, along with using method stubs to declaratively label static memory allocations, allows us to leverage C# against the entire CPU without leaving the comfort of our C# IDE.

The project is definitely still in an infantile state. We need to write an entire runtime support layer so that we can embed a JIT. Remember: think assembler – unless we write memory management, there is no: object foo = new object(); And unless we write type information handling, there is no: if(foo is Int32) {…}

So, in order to stick to our C#-only motif, we have to use a restrained form of C# (that is, no heap types, no type information, etc.), to write a runtime, that the rest of our kernel can be AOTed against. This means we also have to emulate parts of the Mono/.NET corlib. Eventually, our goal is to use our AOT as the JIT that will get embedded in the kernel.

People tend to notice that SharpOS developers are not scared of chicken-and-the-egg problems. If you understood the last couple of paragraphs, you'll understand why.

Currently, the three most active SharpOS members are all doing very different things. xfury is working on porting/emulating common Unix utilities in C# - because we will definitely need them when the time comes. DarxKies, being the lead on the AOT part of the project, is working on writing runtime support so that we can start embedding fully managed code into the kernel (but at that point, still AOTed). And yours truly, has been working on a demo command-line shell to grab the attention of passer-bys, like you.

Here is a screen-shot of the beginnings of fruits of my labors (see the yellow text at the bottom):

You'll notice I had to type the "version" command in twice. Frankly, it's very much full of bugs. But considering our memory management code was just a temporary satisfaction of a complex need, and I had to write a ton of dynamic string handling code from scratch, it's pretty impressive, right?

Within the next few weeks, I'll hopefully iron out the rest of the bugs and code a handful of commands so that we can put out a first release.

Each of the commands is stored as a structure consisting basically of a string pointer for the command's alias, and a function pointer that points to the function that does the work for the command. I know, who thinks of string pointers and function pointers when they think of C#? But when you don't have class instances, and you don't have Runtime Type Information, what is an OO programmer to do?

Like I said, we are in baby stages. (And in need of lots of help!) But hopefully, the screenshot will help get those juices flowing…