Source Code Adventures

Yup, you read that right. This one comes with a big disclaimer. It’s been tested and works but don’t read on if you are squeamish.

The set-up is like this:

We have an application that is deployed via XCopy and a second team in another city who is watching this with a hawk-eye. They have a stable system and don’t want changes that they are not absolutely certain they want. Lots to complain about in this kind of scenario, but that’s what we have to work with.

Now a requirement landed on my desk to change a piece of functionality without sending them a new version of the package (which is used everywhere) that it resides in. Fortunately, this unit (call it One) is only called from one other unit (call it Two).

Unit Two in turn is being used from the main executable – a very tiny little thing that basically loads all the runtime packages that do the actual work. So our hawkish friends are happy to take a change to the main executable, but not to this core package.

My first thought was to add the new functionality to a copy of One, which can be called directly from the executable and be folded back into the core package for the next major version of the application.

This seemed fine, until I realised that the object in unit Two is globally accessible and referenced from all over. Lots and lots of callers that cannot be recompiled. Yikes.

After the laughter gave way to despair, I finally decided that this calls for a hack. You know, the kind of thing you always sagely warn others against. After all, exactly how else do you change the functionality of Two without recompiling it?

Let me rephrase that. How do I call my new code in One, while leaving the internal state of Two identical to how it would have been if I’d used it to call One?

So what is an object anyway?

In Delphi, as you certainly know, object variables are really pointers to the actual objects, so MyList: TList creates a new pointer variable to which you can assign the memory address of a TList object. Some programmers prefer the word reference simply because Delphi handles the dereferencing of the pointer on your behalf, but a pointer is a pointer no matter what you call it.

So what does the object pointer point to exactly?

Well, it points to a record structure. Seriously. The first field of this record structure is a pointer to your virtual method table. Incidentally, if you declare a variable of a class reference type (TClass, TPersistClass…) that is also actually a pointer to the VMT. And if you call ClassType on any object variable, you also get a pointer to the VMT (typecast to a TClass – a pointer is a pointer, right?)

The second field in your record structure is the first field of your class. So, the hypothetical

TMyObject = class
private
  ID: Integer;
  Age: Integer;
  Income: Double;
end;

is identical to

TMyObject = ^TMyRec;
TMyRec = record
  VMTPtr: TClass;
  ID: Integer;
  Age: Integer;
  Income: Double;
end;

Before the flame war starts, let me point out that these two declarations are different in semantics, purpose and the warm fuzzy feelings they give to programmers, but they are identical in memory.

I can get more into this in a future post, but the thing to note here is that our object variable is in the end just another block of memory with a predictable layout. So if I needed to change the internal state of my object, I could do it without paying attention to scope identifiers, methods, implemented interfaces, or the weather.

So I could create a second class with my new functionality as long as the in-memory position of the fields I want to manipulate are the same.

TMyNewObject = class
private
  FID: Integer;
  FAge: Integer;
  FIncome: Double;
  FMonthlyWage: Boolean;
public
  procedure CalculateIncome(ID: Integer; IsMonthly: Integer);
end;

Notice that I have added a field to my new class as well as a method that can implement my new functionality. I’ve kept the other fields the same but could have changed them in any way that would leave the positions the same for those I use in TMyNewObject.

So now my executable can create an instance of TMyNewObject, call CalculateIncome and… then what? Well, remember the whole thing about the TMyObject instance being globally accessible? I need to find a way to assign the values stored in TMyNewObject back to TMyObject without being hindered by the private scope of those fields. But an object is just another block of memory, right?

To copy memory from the one to the other, I need three things:

1. Two object instances, namely Obj: TMyObject and NewObj: TMyNewObject.
2. The in-memory size of the data used by the TMyObject instance.
3. A function to copy from NewObj^ to Obj^.

Requirement 1 is easily met, requirement 2 is met by calling Obj.InstanceSize and requirement 3 is met by using the following:

Move(Pointer(NewObj)^, Pointer(Obj)^, Obj.InstanceSize);

And that’s it! Well, sort of.

This will work just dandy as long as you use only simple types like Integer, Char, Double or Boolean. It does not, however work for managed types like strings or referenced types like objects. And it has a little type confusion issue as well.

String, interface and dynamic array types are managed types and if we just copy NewObj’s data over Obj’s data, we lose Obj’s pointer to this managed memory which means we leak memory. Also, and possibly worse, the memory manager has the impression that only one reference to your managed object exists, so it will free that object as soon as you free either Obj or NewObj. If you then free the other one, you’ll likely receive the dreaded “Invalid pointer operation” message.

A similar issue exists with any pointer or object types, because you lose the pointer to Obj’s data.

The solution here is to not copy the memory from NewObj to Obj, but rather to swap it. You know, like switching the values of two integer variables:

C := A;
A := B;
B := C;

We’ll put this in a reusable function. We’ll also let this new function automatically ensure that it only copies the memory the two objects have in common – so the lesser of Obj.InstanceSize and NewObj.InstanceSize.

The type confusion issue I mentioned relates to the VMT pointer. Remember I said that is actually used for the class type? So if you copy the entire memory block from NewObj to Obj, you are actually changing the runtime type. Certain constructs will still work as expected (confusingly, the is-operator is just fine) but others return type info from TMyNewObject (like ClassInfo or ClassName). Worse, your virtual methods could be all messed up.

So instead of casting like I did in the Move-call above, I’ll use this inline function to skip the VMT pointer:

function GetObjData(Obj: TObject): Pointer; inline;
begin
  Result := Pointer(Integer(Obj) + SizeOf(TClass));
end;

And then we get a perfectly generic function that doesn’t leak memory or cause pointer errors:

procedure SwitchObjects(Object1, Object2: TObject);
var
  Buffer: Pointer;
  BufSize: Integer;
  BuffOffset: Integer;
begin
  BuffOffset := SizeOf(TClass);
  BufSize := Min(Object1.InstanceSize, Object2.InstanceSize) - BuffOffset;
  GetMem(Buffer, BufSize);
  try
    Move(GetObjData(Object1)^, Buffer^, BufSize);
    Move(GetObjData(Object2)^, GetObjData(Object1)^, BufSize);
    Move(Buffer^, GetObjData(Object2)^, BufSize);
  finally
    FreeMem(Buffer);
  end;
end;

So for my nightmare scenario that I described at the start of this post, the solution turned out to be simple. Instead of directly calling the needed method on my global object, I passed it as parameter to a function that:

1. Creates my new functionality object.
2. Calls the new functions.
3. Switches content with the global object.
4. And returns.

This is about as clean as the hack can get, I’d say. Obviously, the new functionality will be folded back into the core package for the next major release of our software, but for now everyone is happy.

What truly ugly hacks have you had to apply?