Warning to those about to re-invent the wheel

I have to warn those of you (or us) that find them self re-inventing the wheel.

First, and we all know this, that you should not re-invent the wheel unless it is absolutely necessary. Often there is a performance criteria that causes the wheel to be remade.

I have dealt with four major "new" wheels that I vividly recall.

1) Memory Manager
2) WinMac - a port of the Win32 API to the Macintosh
3) XML parser
4) The Gadget Library, a replacement for WinForms controls, layout, and UI containment

All of these have commonalities.
1) Performance was the driving reason
2) Replacing well known and highly used functionality
3) Complex code to give the performance increase
4) One or two developers owning the code


The Memory Manager

The new memory manager allowed a product who's core was indexing and retrieval to be extremely fast on even Intel 386 based machines. Without such speed the product could not have gained the user base that it ultimately maintained.

The growth of the company and the spreading use of the product allowed for the development of a Unix version. That is where I came into the picture. Since I was a Mac programmer I also did some Unix development, or another way to read this was I was a Win'tel hater. Youthful convictions run deeper than reason.

I looked at porting the memory manager to Unix and found that the memory management on Unix was faster, so there was no need for the port. However I had to port the interface of the new memory manager so that I could reuse the "C" code of the product. So a bit of warning, performance issues and the resulting optimizations may not, most likely will not, be of value to cross platform code.

Finally the memory manager itself was the bottleneck. It was so in two ways. First, as new versions of Windows came out and new compilers came out the native memory manager became faster than the custom one. Second, since the memory manager was developed by one person he could not keep his custom memory manager up to date, he was a bottleneck.

Eventually the memory manager was removed. Was it necessary? Yes, it was. The company would not have grown to the point where the memory manager became obsolete if it hadn't have sold well allowing the company to stay in business and grow.

The memory manager should have been architected differently. The issue was it had its own procedure calls completely different than "Stdlib". If the interface would have been the same as the standard memory manager then it would have been a simple swap out. So, when replacing something like the memory manager, stick to the interfaces so that it is a compile time or link time switch that uses the desired memory manager.


WinMac - a port of the Win32 API to the Macintosh

A good friend of mine took the task of porting a Win32 application to the Macintosh. He is an excellent programmer and therefore there is no coding task that he will not consider as the correct solution to the problem. (I on the otherhand will say, 'Yes it can be coded, but should it be coded?')

To make a long story short he decided to get the Windows code to compile on the Mac without any modification to the code. Sounds good to management, one code base, all new Windows functionality is immediately available on the Mac. In those days no one even considered if it was legal, those types of legal issues had never been spoken of.

Well, he got behind, had no one to bounce ideas off of, and so they decided to hire someone to help. That is where I come in. The pay raise was around 15%, who could say no.

There were some big problems.

1) The WinMac library was a subset of Win32. It only contained a port of the Win32 calls that were made by the Windows product.

2) Macintosh OS 7 (8 and 9, and those before) is an event based model with one main event loop and cooperative multitasking. (I have dealt with crazy Mac programmers that coded up multiple event loops! Shame on them. SHAME.) Windows is a message based UI with crazy message pumps and all of those message id's, etc.

3) No specification of Win32. Just run it, use the event spy, and figure it out. There was documentation on how to developer for WIn32, but not a specification on how Win32 was developed.

4) Bugs in Windows.

As we developed on the single code base we quickly found ourselve's downstream of the changes made to the Windows product. We would get the code compiling and then a few days later get code again to find that some developer called some new Windows API that we hadn't ported yet. We ended up porting the vast majority of the entire Windows API.

Also, we found many bugs in Windows and the order of messages. The documentation would state a certain series of messages but the event spy showed at runtime that Windows did not do it that way. So, we had to mimic even the bugs.

Finally my friend got into a workplace polictical conflict with management and left me all alone.

Lesson learned, it is better to port ideas than it is code!


The XML parser

Once again performance was the issue. I do admit the developers I work with have a good vision for the future and when XML was very young the idea of using it for making calls to servers through http was leading edge. So, as SOAP is coming into existence the company has moved far ahead. Standard emerge and the team keeps the code up to date as best as they can. However, the XML that was used to test the custom XML parser was unique in that all attributes where in double quotes. No one, by habbit or dumb luck, ever passed xml that had attributes in single quotes. Well, that is where I come in. I had started using XML later and had used the libraries that came with the IDE. I spent several days trying to debug code that was an http call passing xml to a server and I kept getting exceptions. My XML was valid, I had checked in many times, had others look at it, etc. One of the "old timers" noticed the single quotes and said, "There's the problem".

As the company grew rapidly the new/younger developers started writing SOAP implementations and now there were both worlds in existense at the same time. At first SOAP was slower and those that used it got chastized by the old timers for not stepping up and learning the proprietary solution. But, eventually SOAP gets the performance it needs.

Lesson learned here is that if you are have a replacement for something in which there are standards then you have to keep up with the standard. The personel working of the code is the bottleneck. Typically an update in a standard is significant and you get it dumped upon you all at once. Therefore you find yourself picking and chosing and therefore you are now noncompliant.



The Gadget Library, a replacement for WinForms controls, layout, and UI containment

This is my most recent experience. Performance was the issue, that and the inability to create thousands of Controls dynamically, to draw all of these controls to offscreen bitmaps, or to visuall scale/zoom all of the controls.

Because I have replaced WinForm's controls, completely replaced them I find myself the bottleneck to getting in new functionality. Imagine just one guy owning the WinForms classes for Microsoft. Each bug, each new feature, each change is huge. The amount of code to keep in one's head is tremendous as well.

The issue is that UI components and their abilities are a commodity now. When I developed a gadget it was coded to do just what was needed and nothing extra. YAGNI was the driving mantra. Users are unaware that the Gadget library exists, it all looks like Windows to them. So, they say, "We would like it do X". "X" would be easy enough if I was using WinForm's controls. The users expect it and management doesn't necessarily see the difficulty in asking for it. But it is starting to wear me out. Too much code, too much responsibility, to much complexity, just too dang much.

The lesson learned here is when replacing a UI commodity, you should fully replace it in most cases. Fully means, full functionality implemented, all of the behavior of the original.

Final Thoughts

There are some items of interest as well, based upon my experience. When I was an undergrad at BYU I took several upper level classes. One of which was a GUI class by Dr. Olsen. I don't think that man ever liked me. In that class I wrote a GUI system on my little Mac SE. We were constrained and could only use the basic draw functions, like draw rect, draw string, etc. I made windows, controls, scroll bars, drop down menus, and other GUI components from scratch.

I would go home for the summers and work on my Father's dairy farm (which also included about 200 acres of alfalfa). I would purchase the computer science text books for the classes I would take in the fall and read them and work through the end of chapter items. During one summer I wrote a new Macintosh GUI system where I developed a language for describing a UI. I wrote a parser and code generator for that language. This was around 1986 or so.

Then I helped write that WinMac monstrosity. With these experiences in developing GUI environments I was at least able to write the Gadget library I use with WinForms development. The point is the path I had taken, or better yet been herded down, gave me the knowledge to be able to make the Gadget library which is extremely fast and meets the performance requirements. The hard road, the big tasks, the complex assignments, all have helped make me what I am capable of doing today.


Geoff