Category Archives: General

SourceMod Project Changes

Over the past three weeks we’ve made some big changes to SourceMod in terms of managing the project.

I have pretty much nothing but praise for this tool. It’s easy to set up and configure. In about a day I had replaced SourceMod’s terrible and messy Perl scripts. And it looks very pretty.

We have a separate build server with slaves for Linux and Windows (Window is running under VMWare). The master lives on the web server. The version control system (was Subversion, is now Mercurial) notifies the master of source code changes, and the slaves build SourceMod and push snapshots back to the web server. It also manages the symbol server.

Our scripts for buildbot live in tools/buildbot under the source tree for all who are curious.

I’ve come to love Mercurial, despite its idiosyncrasies, from using it at work constantly. The distributed nature makes everything much easier. I can keep many copies of the repository around on different computers and share changesets in between them and other people. I don’t need internet access to push changes and I can juggle many patches at once.

We started to notice a problem in SourceMod. We had two branches: 1.0, and 1.1. When we pushed bug fixes to 1.0, there was no way to share these back to 1.1 while maintaining change history. Mercurial inherently solves this. Subversion 1.5 supposedly has merge tracking, but by the time it was out our minds were already made up. Mercurial’s distributed nature is awesome.

I haven’t learned how to use patch queues yet (mqueue), but I’m being encouraged to at work. Our Mercurial repositories are here and I’ve written a tutorial (akin to our CVS and Subversion tutorials) here.

I originally chose Flyspray over Bugzilla because it looked nicer and I could understand the permissions system. Eventually I came to realize that Flyspray was easy to outgrow, and I came to like Bugzilla more and more from using it at work.

Flyspray is good, and probably suffices for small projects. But it’s buggy. Our install was breaking for some reason, we couldn’t search for users anymore (every user would return “not found”). Marking a project as inactive would let normal users edit bugs still. There seemed to be no way to have group-specific permissions, which meant we couldn’t have private or internal projects. The UI didn’t work on mobile devices (this is probably the fault of poor AJAX implementations). You couldn’t refresh a bug after posting a comment, because there was no link back to the bug.

Bugzilla has a much more complex UI but there are a few things that sealed the deal. Aside from addressing the problems Flyspray had, its authentication system was a lot more flexible internally. I was able to write a simple authentication plugin for vBulletin. The ability to watch users makes automated CC’ing much more flexible. Now anyone can listen for all SourceMod bugs, whereas previously only the project owner could.

The attachment system is a huge win. Attachments can be obsoleted and they all live on the same section of the screen, near the top, rather than sprinkled throughout comments. It can show diffs for patches which is really nice. You can request peer-reviews on patches which is something Mozilla does extensively.

If anyone’s looking to convert from Flyspray to Bugzilla, you can have my hacked-up Perl script. It uses an intermediate table called “mvtab” (type varchar, old_id int, new_id int) as a temporary work table. You need to create this yourself. You also need to manually import products and components (components must have the same names as categories). It’s really a mess but it got the job done, so play with it at your own risk. (tack-on, since the original import didn’t get e-mail settings right)

And here’s the Bugzilla vBulletin binding: (goes in Bugzilla/Auth/Verify). You may need to change config files somewhere to use it.

SourceMod’s JIT Opened, Performance Ideas

Originally, we had a few motivations for keeping SourceMod’s JIT closed source. They are neither here nor there, but we’ve decided to open it. I’d like to talk a bit about what it is and what it isn’t. If you’d like to see the source, it’s in sourcepawn/jit/x86.

A JIT is a technology that, “when it’s needed,” compiles code from one form to another for optimization. For example, the Java JIT compiles Java bytecode to native assembly. There are three levels of granularity in a JIT:

  • Whole Program. The JIT compiles the entire program in one go (actually an “ahead of time” compiler). No granularity.
  • Methods. The JIT compiles functions one at a time as they are needed. Coarse granularity.
  • Tracing. Can compile any path of code anywhere in the program. Fine granularity.

Tracing JITs are new and still seem to be experimental. Traditional JITs, like Microsoft’s and Sun’s, compile functions one at a time. SourceMod’s JIT is a whole program compiler. We chose this route for a few reasons:

  • It’s easy.
  • We’re not concerned about loading/compilation time since Half-Life servers take forever to start anyway.
  • Even so, the cost of computing optimizations on a SourceMod plugin is extremely cheap.
  • There is no performance gain from tracing because Half-Life runs on good processors and all types are statically known.

Why is optimizing Pawn code so easy? It has a simple, two-register design internally. This has a number of really nice implications:

  • The generated patterns are simple. Peephole optimization in the source compiler does a good job at reducing expressions.
  • The two registers that represent machine state can be easily mapped to processor registers, since nearly every processor has two scratch registers.
  • Types are static and optimal code can be emitted.

Therefore SourceMod’s JIT is “dumb” for the most part. It performs a massive peephole “search and replace” of every Pawn bytecode to native machine code. Where it wins is that the assembly is highly handcrafted to the x86 ISA, rather than piped through a processor abstraction layer. faluco spent a lot of work optimizing moves and stores to reduce things like pipeline stalls. A pipeline stall is when one instruction depends on the result before it. For example, if x86 sees an address coming up, it tries to pre-fetch the final computed address onto the address pipeline. This is why LEA is called a “free” instruction on x86 (because the computed result is “already there”). If the address computation requires a prior instruction, the pipeline will be stalled.

SourceMod performs some other more general optimizations. It optimizes for space by emitting short instructions when possible. Certain function intrinsics are implemented in inlined hand-crafted assembly (like the GENARRAY and FILL opcodes). Native dispatch is entirely inlined down to assembly.

A special amount of attention is given to switch cases. If all case values are consecutive they are optimized down to a jump table rather than a series of if statements. The sequence can start at any number and end at any number, as long as no number is skipped.

Since Pawn doesn’t have types, the JIT performs some peephole optimizations on certain natives. For example, if it sees the ‘FloatAdd’ native, it will optimize the code down to FPU instructions. This is a huge bonus because native dispatch is expensive (the VM’s internal state must be cached and then restored). This specialized peephole optimization occurs mainly on float natives.

The JIT maps Pawn’s two registers to EAX and EDX. This ends up being really nice, as they’re the two scratch registers used for lots of important things on x86. For example, a MUL instruction uses both, and many of the ALU instructions have shortened forms for EAX. The unfortunate side effect is that stack spilling can be common, but the Sethi-Ullman Algorithm shows that two registers will suffice for binary expressions.

The SourceMod JIT gets the job done. In terms of design considerations, it’s not perfect. Its optimizations are primitive, and it assumes the source compiler will do most of the hard work. But in the end, if fits the bill well. Pawn doesn’t have much complexity, and the scripts are fairly small.

I do have some future plans for the JIT. Some easy optimizations are that more float natives can be optimized away in the peephole pipeline. Similarly, if SSE/SSE2 is detected, faster instructions could be emitted instead.

With a little help from the compiler, we could completely eliminate a large class of useless performance losses in SourceMod. Pawn scripts use “local addresses” that are relative to a base pointer. With a little help from the compiler, the JIT could eliminate local addresses completely. This would greatly improve generated code performance and turn LocalTo* calls (which extension writers should hate by now) into nops. There are other implications too – calls from plugin to plugin would become direct calls instead of slowly marshaled through PushCell() and GetNativeCell() and whatnot.

Lastly, natives are expensive. It takes 28 instructions to jump into native mode. Unfortunately, calls like IsClientConnected() and IsValidEntity() can’t be implemented without native code. A possible solution to this is to introduce a few very specialized “native” opcodes into the JIT. For example, a check for IsClientConnected might look like this in an extended Pawn bytecode:

  push.s  0xC     ;push the first parameter (the entity index)
  getuseraddr 1   ;offset of &g_Players address in context::user[] array
  vcall 5         ;call function at virtual index N, GetGamePlayer()
  vcall 2         ;call function at virtual index N, IsConnected()

The JIT would then compile this directly into the plugin, as if it were a stock. This would effectively let anyone write very fast natives in an abstracted assembly language, without incurring any native overhead. It is of course a lazy solution based on Pawn’s limitations, but definitely a decent attempt at gaining performance for very little work.

For now, I will likely back away from other lacking things in Pawn (like CSE, liveness analysis, optimizing register usage, etc). The principle reason being that the other proposed optimizations are likely to get more bang for the buck at this point. The fact that the JIT is hardcoded to assume EAX/EDX as the two storage registers means such extensive optimization would warrant a complete redesign for a dynamic register allocator. We wouldn’t be able to justify that until we freed up some of the registers required by Pawn (like the DAT and FRM registers for the local addressing mode, see earlier idea).

Why I’m not an ECE Major

I’ve always felt that Computer Science is somewhat of a liberal arts major compared to the hardy breed that is electrical engineers. So I’m taking an introductory ECE course to satisfy my own curiosity.

Today we had a simple lab demonstrating digital logic. I prepared part of the circuit on our breadboard and my lab partner finished the rest.

Unfortunately, nothing worked. I spent about fifteen minutes rewiring and pulling apart the circuit until nothing was left but the breadboard and a single logic gate. Even then I still wasn’t getting the output I expected.

I said, perplexed, “Well, I have no idea what’s wrong.” My lab partner stared at the board for a few seconds and then remarked, “Does it matter that you have the power source plugged into the GND line instead of the power line?”

I had pulled apart our entire circuit because I had forgot to plug in the power, once again reminding me that I should keep a safe distance from anything resembling hardware.

Asking the Wrong Audience

One thing that happens quite often in our forums and IRC channels is that someone (invariably new) will enter and ask a question completely unrelated to the topic. Typically, about competing projects or projects on the same platform.

For example:

  • Topic: AMX Mod X. Question: “How do I use EntMod?”
  • Topic: SourceMod. Question: “How can I use EventScripts?”
  • Topic: SourceMod. Question: “Can someone help me with a Mani issue?”

Onlookers tend to reply the same every time: “Visit their IRC channel or website for documentation. This channel/forum is unrelated.” The person asking the question then says, “I tried, but no one there is replying.”

The person is trying to keep their question alive in the off-chance someone random will know the answer. If you do get a reply, all the best. But if you don’t, you have to realize that nothing you’ve said changes the fact that you’re still asking the wrong audience.

Would you ask your English professor a math question because your math professor isn’t in his office? Would you hire a dentist for a plumbing job because your plumber is out of town? Of course not, that’d be silly. But sometimes, if the person doesn’t get a reply, they will become aggressive or insulting. For example, “Thanks for no help whatsoever!”

If you’re knowingly asking the wrong audience, don’t throw a tantrum when you don’t get your answer, and don’t get annoyed when you’re told a better place to look.

I’ve Dumped TortoiseSVN

I hate TortoiseSVN. The interface is great and it makes working with SVN a lot easier, but it’s so buggy it’s unbearable. Just a few of my complaints:

  • It’s slow. After a fresh install of Windows XP and TortoiseSVN, it completely locked up Explorer while it attempted to traverse a massive repository I had.
  • It causes weird explorer behavior. In particular, it feels like the screen is getting redrawn way too much.
  • TSVNCache is a horrible idea, or at least horribly coded. I often find that I can’t eject removable media because TSVNCache is holding file locks on random files, even if explorer is closed. Why? I have no idea. The only solution has been to kill it.
  • Sometimes, files randomly won’t have the right overlay for their status. Sometimes refreshing works, sometimes you have to exit (or even kill) explorer. In worse cases, TSVNCache must be killed, and in the worst case, you have to reboot (I’ve only had this happen once).
  • On my Core 2 Duo with 2GB of RAM, TortoiseSVN adds noticeable delay to explorer. It takes a split second longer to draw everything. The bigger (or deeper) your repository gets, the longer the delay seems to be.

So, TSVN is buggy and slow. I now use the command line on Windows instead. Getting this working with SSH keys is a bit of a chore, but here’s what I did:

  • Download the Windows PuTTY installer (I installed to C:\Program Files\PuTTy). Click here to get the exact package I did.
  • Get the latest Subversion package for Windows. I chose the binaries compiled against “Apache 2.2” — the exact package was “”
  • Extract the zip file to somewhere. I chose C:\Tools and renamed the main subversion folder to get C:\Tools\Subversion.
  • Go to Start -> Settings -> Control Panel -> Advanced -> Environment Variables.
  • Edit the “Path” variable under “System variables,” and append the following string to the end, using your subversion’s bin path: ;C:\Tools\Subversion\bin
  • Add a new variable under “User variables.” Call it “SVN_EDITOR” and set it to some text editor you like (mine points to C:\Windows\notepad.exe).
  • Open notepad, and open the file “Application Data/Subversion/config” where “Application Data” is the folder under your user account (i.e. “C:\Documents and Settings\dvander\Application Data\…” or just “C:\Users” on Vista).
  • Under the “[tunnels]” section, find the commented line that looks like:

    # ssh = $SVN_SSH ssh

    Replace it with:

    ssh = C:/Program Files/PuTTy/plink.exe

    Or something similar depending on where you installed plink.exe.

  • Load pageant from the same folder plink is in (C:\Program Files\PuTTy\pageant.exe for me).
  • Load your SSH keys into pageant.

If you don’t use SSH keys, or don’t know what they are, see this article.

You can now run subversion from the command line. Examples:

svn co svn+ssh://[email protected]/svnroot/sourcemm sourcemm
cd sourcemm
svn update
svn commit

If it doesn’t work, make sure your environment is updated. For example, start a new cmd session rather than using an old one.

Note: I never had any problems with TortoiseCVS.


This is my personal continuation of the discontinued SourceMod DevLog. You can read all of my old archived articles there. If I have time, I will move the good ones here.

A new article will appear every Monday at 10:00AM CST under the “Articles” Category. I may slip in other random things in-between since this is my personal blog now.

I’m sorry but you have to re-register to comment. Please PM me if you have problems with registration or comments.