Migrating to 64 bits

Sooner or later you'll find at the crossroads of deciding whether or not switch to 64 bit. If you want to be future proof, do it!

Migrating our engine to 64 bit has been quite the ride. Here are some tips and recommendations:

* Make a new configuration off your 32 bit one. Then change the code generation to be x64.
* Temporarily remove all 3rd party libs linking with your project, and #ifdef the code that uses it out with some define. This is usually the most boring part; the goal here is to be able to compile/link first. We'll fix the runtime issues later.
* Once your solution links & compiles, try to run it for kick's sake. And start fixing the errors as you go.
* If you load binary data directly off disk, you'll have to work on each tool that generates binary data and make it work for 64 bit (or gasp! Both 32 & 64 bit!)
* The biggest issue we found during our conversion was pointers and padding. I'll discuss that in another post.
* You might be able to live with having 32 bit tools that generate 64 bit data. The usual problem with this is if you have a struct/class which you write out directly to disk, but there are ways around this.

Droid X Freezing

So I started getting weird/random glitches on my Droid X, where the phone would just freeze. I could press the top button & turn off the screen, but when pressing it again or the home key to unlock, sometimes the bottom row of physical buttons would light up a bit and then turn off, or it would just not do that and keep the screen off.

Unfortunately I was out of the country so I had no internet or data access (I suspected it had to do with setting it in airplane mode for the first time), so I tried removing the SD card and voila! It stopped freezing.

But I could take no pictures, or read my kindle books offline (or play all the games in my sd card!) :(

Coming back I looked into this online and did the following:
1) Backup SD card to PC (worked!)
2) Format SD card from PC (worked!)
3) Copied some folders from backup to SD, put the card back into the phone, turn on and wait for hang...

So did 3) until I only missed the Music folder with all my MP3, and currently the phone works ok! I now have to go through all the files in the Music folder (about 6 gb) and check for weird issues...

So definitely a case of either a corrupt filesys or corrupt file (or an MP3 file with bad tags?).

#Fail

After spending quite some time trying to find a solution, gave up and asked stackoverflow...

Switching phones

Ok, I'm going from an old BlackBerry 8703e, to a Motorola DROID X Android Phone (Verizon Wireless), hopefully I'll see the Android Difference ;)

Finally here! Installation pics of my car stereo

It took a me a while to find the pictures of how the Sony MEX-BT3800U got installed in my Mitsubishi Endeavor, but here they are now!

Pull from the bottom left side a little bit:

 Now from the top right side:

Now from the top left:

At this point it has detached from the dashboard, and you can now pull from the top a little bit:

So far so good; now we have to remove that pair of screws you can see on the top, and with it we detach the cover of the clock/fm lcd:

You see those 2 black wires & the one with foam? Now if we look at this from the top, you can see they are attached to the grey metal cover, so manually unplug each one of them from the white plugs:

Ok, so after that, we can fully remove the grey panel!

Missing from this picture is the old stereo, which would be in the center slot; you'd proceed to remove the front 2 screws to take it out, and the ground cable.

To attach it back, follow the steps in reverse, and push it in the end towards the dashboard.

Disclaimer: You may burn your vehicle electronics! Detach at your own risk!

And the final product:

Super Mario All-Stars

Waiting for it... Free shipping!

Link...

Motherboard upgrade

Seems I've been getting a lot of random freezes on Win 7 x64; I'll blame it on the motherboard, a GA-P35-DS3L, so I'll be switching to a GA-EP45-UD3L instead. At this point who knows why it broke down...

Waiting for Metroid...

I've got a preorder ready! Let's see what Team Ninja do with it!

Installed...

I will take a nicer picture. This was taken quickly last night with my cell phone.

The cable you see there is the USB connector for the iPod.
I will be adding the external bluetooth microphone (so I have to tear it apart) so I'll be documenting that with (nicer) pictures :)

Mitsubishi Endeavor 2005 LS Aftermarket Stereo Installation

I wanted to have iPod support & bluetooth to my 05 Endeavor, but it seems the stock stereo has no way of hooking up an aux-in. I thought of getting a Garmin Nuvi 765T that has bluetooth & aux-in and use it on the fm radio... But I guess I figured I *could* install an aftermarket stereo all by my self (famous last words).

This was a pain to do! Seems all the online info is mostly for XLS owners which have the upgraded Infinity stereo system. I have the LS trim, so the option was to either make a new dash panel (yeah right, I can't even cut with scissors in a straight line!) or to use the lower pocket on the dashboard. That couldn't be too bad, could it?

In the end, success! I'll describe the issues I found and some pics...

Showing std::vector in the debugger (continued)

Seems I found an even better way to show the container elements in the debugger: just add a watch for the address of the std::vector instance!

For example:
std::vector <> oList;
oList.push_back( -1 );
oList.push_back( 6 );
__asm int 3 /// Breakpoint here...


Add a watch for :
&oList


And it expands to (correctly sizing it!)
oList[ 0 ] = -1;
oList[ 1 ] = 6;

Showing std::vector elements in the debugger

Some std::vector instantiations do show up the members as arrays on Visual Studio 2005, some don't; if you need to access the member, you can use the following watch:

oYourVector._Myfirst[ nElement ]

Time...

Definitely the most valuable and scarce resource. It's hard (with a 2.5 year old toddler and a 1.5 month old baby) to have enough time to:

• On DS: Play Exit, Picross and Etrian Odyssey
• On PS3: Play Ninja Gaiden Sigma
• On Wii: Play MadWorld and Punch-Out
• On Xbox 360: Play Silent Hill 5, Gears of War, Condemned 2, Dead Space on Hard mode
• Play Classical Guitar & Electric Guitar
• Play Piano
• Write an MP3 decoder/player from scratch
• Write an HLSL optimizer/extender from scratch
• Learn Scheme
  

But there will be time someday...

Happy 15th Anniversary...

Super Metroid!

This nice piece on Gamasutra reminded me of it. I'm getting old! I 'borrowed' it from my friend's brother when I was 14... I think it mysteriously never got back to its original owner ;)

And Code Too...

Of course code also needs to be elegant, concise, clean, self-documenting and efficient.

My previous post was directed toward performance critical systems, those systems where you know it's the make-or-break 60 fps framerate...

Good class design (maybe even design patterns) is a characteristic of a good programmer. But maybe the best programmers can adapt and produce the right solution for a problem.

In other words, Empty your mind, be formless. Shapeless, like water. If you put water into a cup, it becomes the cup. You put water into a bottle and it becomes the bottle. You put it in a teapot it becomes the teapot. Now, water can flow or it can crash. Be water my friend.

So much to learn, so little time to do it...

Not Code, but Data...

I've spent countless hours trying to come up with clean, elegant, concise and readable systems/APIs. Hours spent trying to see if a Design Pattern will help me, refactoring code, reading about best practices, etc.

But one day I had a Zen-like revelation.

It's not about the code, it's about the data!

See, all a computer/program does is transform and move data from one format to another. All we do when we try to come up with fancy interfaces are just ways for us humans to understand how the computer/program will transform the data.

Of course, you still need to program the APIs in a nice, clean, elegant and readable way, but keep in mind that the end product is just transforming/moving data. Streams of bits and bytes get split up, operated on, and joined into new streams of bits and bytes.

So in games programming/engineering, you need to figure out what data you need. Come up with a transformation function, and then clean up/refactor/humanize the code without losing performance.

I think that's the meat of games programming.

Shader tip

Tired of recompiling shaders all the time to try different combinations or code branches? Use this little hack in pseudo-code for selecting values: (I guess this was more useful on pre-shader model 3.0 hardware)

float4 g_vControl = float4( 1, 0, 0, 0 ); // Default to Branch1

float4 MainPS() : COLOR0
{
  float fValue0 = ResultFromBranch1();
  float fValue1 = ResultFromBranch2();
  float fValue2 = ResultFromBranch3();
  float fValue3 = ResultFromBranch4();
  float4 vValues = float4( fValue0, fValue1, fValue2, fValue3 );
  float fResult = dot( g_vControl, vValues );

// Do more math with the result...
}


Then on your main C#/C++ code, you can just set g_vControl to ( 1, 0, 0, 0) for Branch1, ( 0, 1, 0, 0 ) for Branch2, etc.

Of course, on release code you should remove this, or set g_vControl to be static bool so the compiler will optimize out the remaining branches. But for developing it's quite useful.

Also, you could blend the values: g_vControl = ( 0.5, 0, 0.5, 0 ) for example to blend Branch1() and Branch3().

Now a PS3...

So now I got a PS3. Setup was pretty easy, but I still have major issues with its GUI.

Seems very old fashioned, very cluttered. On one hand it's nice to see the past menus you came from, but it just makes it look very busy.

Streaming with TVersity "just works". I love that it can play VOB files directly, whereas the 360 seems to have issues with them. It would be even more awesome if I enable subtitles from the VOB file (maybe it's not inside it)!

Blu-Ray and DVD playback is easier than the 360; I love that it can autoplay the movies when you turn the console on, whereas on the 360 you have to browse to the DVD or HD-DVD drive on My Xbox, then click on Play Movie.

It seems to run way hotter than my 360 for some reason, maybe the 360 has one of the newer chips that makes it run cooler?

I measured 106 degrees F inside the audio cabinet with the door closed, wow! (It accidentally stayed on when the cabinet door was closed...). I'm afraid to run this same test on the 360 :)

The only thing that really irritates me is that I can't play any PS2 games on it. Seriously, what's the deal with that?

I really hope Sony's working on a PS3 software emulator. There were so many games I almost bought for PS2, but thankfully I read the PS3's box... Not all of us had a PS2, Sony!

Don't eliminate the programmer...

I came across this very interesting post by Scot Westfall (SlickEdit's blog) "Eliminating the Programmer":

Here are some great points:

• "[...]programmers have skills and abilities other than just their knowledge of programming languages."
• "[...]Regardless, I’ve noticed that programmers are often more capable in many ways than their non-programming coworkers."
• "Programmers think more logically. Working through if-then-else conditions is a core capability for any programmer. While working with business teams on requirements, I have often run across cases the where same ability was lacking."
• "Programmers have a superior ability to analyze problems and come up with solutions. They excel at analyzing preconditions, sequences of events, and outcomes. Certainly, this is a key skill in programming, but it is also useful in troubleshooting and business case analysis."
• "Another key ability where programmers typically have an edge is the ability to make order out of chaos. I think that’s because the programmer is responsible for creating order within the program."
• "[...]their main talent lies in their ability to analyze, troubleshoot, and solve problems. Code is just the physical manifestation that culminates the thought process of the programmer."
  

Excellent insight into what being a programmer is all about.

Of course,

• "it’s clear that this is not a universal trait in all programmers"

Read it in full, and enjoy!

XBox 360 & Verizon FiOS Networking adventures

As discussed before, I got a 360. So it was quite a bit of a pain getting this to work with my Verizon FiOS network connection, specially because paying $100 for a wireless adapter is kind of expensive, and my wireless signal's bad. Here's how thing are:

FiOS is the Verizon fiber optic service for internet/cable/TV/phone, etc that supplies my home with information and entertainment. The house is (as most of the houses on the US) wired for coaxial/cable all over, at least on each room.

The fiber optic from FiOS gets home from the street into my garage, where they have a hub which converts it to coaxial. This hub also has an ethernet output, which would be awesome and solve all my problems had the house be wired with ethernet. Anyways, so from this hub coaxial is conected to the house's wiring and that distributes it to the rooms.

My desktop is upstairs, and from the coaxial there's a so-called MoCA converter, which convertes coaxial to ethernet through Verizon's router, an Actiontec MI424WR. This is a wireless router too. So I plug my PC from there and use the wireless on the laptop downstairs and the Wii. Signal's pretty bad, too many walls in between (my house has a weird layout).

So the first thing I though is well, if I can get my DLink DI-524 as a wireless bridge, I can then just plug the 360 (which is downstairs too) directly into it and be done. So I looked at the DD-WRT project, but seems my DLink doesn't have enough on-board ram to be reflashed and reused as a bridge.

Just for kicks sake, I got a WRT54GL router and tried it with dd-wrt, and seems the Verizon router has issues with some routing table being too small. I also tried hooking up both the 54GL and the DLink's WLAN port into to the Verizon's LAN port, but that seems to kill the Verizon router.

After looking over pages and pages of info, I found this pretty useful one. So I would have to get new hardware...

Got on eBay and bought the same Actiontec router from Verizon for $20 bucks. Verizon wanted $150 for a new one. Crazy talk.

So days later I got the router.

So I looked at the previously mentioned report, double checked my equipment matched the diagram from the tutorial, and proceeded. I set the range of IPs for the 'main' router, the one where the 360 was hooked up, to 192.168.1.1 through 49, and set the static ip of the secondary (where the PC is connected) to 192.168.1.50, and the range of addresses to give from 51 to 99.

I got an IP for my PC of 192.168.1.51, which sounded good. I could ping the main router (1.1), and the secondary router (1.50). So far, so good!

Then I tried pinging www.yahoo.com, and... no cookie. Tried ipconfig /renew, tried changing settings and port forwarding for http and https ports in the verizon config for both routers, but still nothing.

So then I looked at my PC's ipconfig, and saw this:
IP: 192.168.1.51
Mask: 255.255.255.0
Gateway: 192.168.1.50

After some fiddling with it (and getting angry), I figured out that the gateway was the issue. So I went ahead and changed my PC's IP to be static (192.168.1.51) and forced the gateway to be the main router (192.168.1.1).

Everything works like a charm.

Not bad for somebody who slept through most of his computer networks classes ;)

XBox 360, WMP 11 & Streaming

So finally bit the bullet and got a 360. Just took me a few years to "jump in". What a wonderful piece of hardware! (Ignoring the three red rings of death issues). Getting used to hi-def console is hard, now I can see I might need a new prescription on my eyeglasses.

The new dashboard is nice, feels a lot more like the Wii channels but more 'professional'.

I can also now stream Netflix movies (using Netflix's own 360 downloaded app), PC audio, PC movies and my PC photos.

Which application I did that with?

The first thing I tried was Windows Media Player 11 from my laptop, which has Vista. (Why not the desktop? See a future post about networking woes). It just detected the 360 and the 360 detected the laptop, had to enter a code from the 360 and that's that, I could see/stream the files from the laptop to the 360.

However, I don't want the poor laptop's 160 GB drive being killed from streaming. Nor do I want the wireless connection's speed. I wanted the wired network to my desktop, where the big guns are.

So I tried WMP 11 on my desktop, but it just began to go nuts on my media files. Started changing attributes on my folder.jpg files, adding those AlbumArt*.jpg, and all sorts of crap.

I tried turning the options for WMP to now mess with my library but it just blissfully ignored me. So Google to the rescue!

I went with TVersity instead. Works like a charm! Doesn't mess up my files, streams with no visible CPU hit and monitors folders pretty nicely.

So now, I can get the kid's MP3 played on the 360, so she won't try to put her music CDs inside my home theater and/or dvd player and get them more scratched :)

Yet another reason to justify buying a 360. Hopefully the wife will buy it.

Keep it simple...

A colleague and I were discussing about which were the things that we learned from other programmers which made us better. I went on and on about how I learned to engineer software in baby-steps and how that helped me, but we ended up figuring out what makes great code robust and easy to use:

Keep it simple.

Not trying to over-engineer systems, not trying to hide things inside obfuscated APIs, basically not trying to look 'smart', and giving very simple pieces/components that have thin layers, will make you a better programmer.

If it is hard to explain, it probably is hard to mantain and expand.

At least on my experience, it has been a good rule of thumb. Specially now that I have it on my conscious mind :)

Still more STOP 0x00000024

I still can't track this down. My PC was up and running (and idle) for about 9 days, then suddenly it got the same STOP 0x00000024 error. I have Avast! as my antivirus, and Live FolderShare running all day, to sync files between work and home. And I played BioShock for several days in a row, no issues, exiting the game and saving correctly.

So I had to reinstall Windows on the IDE hard drive again, boot from that installation, which forces a chkdsk on the SATA drive, and then change the boot.ini file to boot on the SATA drive again.

chkdsk showed errors on a the files and folders I'm syncing with FolderShare, so that may be the culprit... Had to restore a bunch of them from backup.

I'm thinking it's a hard drive error, but the SeaTools suite keep telling me everything's good.

Black Textures on BioShock

So BioShock ROCKS. Ok, we know that.

Now for the issues...


I tried the demo back on my 6600GT AGP card a while ago, and figured it couldn't run (or look nice enough) with that card. So I uninstalled it.

Then I bought the retail version on my last upgrade, and had some issues with it.

I did some WinDbg debugging, some cleanup, chkdsks, etc and now the PC stopped crashing.

After a while, 30% of the game started showing with black textures, as if the lighting was messed up. Even worse, some characters (and bots/turrets) showed invisible textures, which made it very, VERY frustrating to play (I am walking happily, then suddenly I start getting my health all drained up. Guess what? Invisible enemy).

So I looked in forums all over, found nobody with my exact problem. Tweaked the hell of all the available .INI files in the games. Until I found a solution buried under a forum:

* Uninstall BioShock Retail
* Install BioShock Demo
* Uninstall BioShock Demo
* Install BioShock Retail

All's good now.

It's really perplexing since I did a full Windows Install after I first installed, so no traces of the BioShock demo would be there...

STOP 0x00000024

So I upgraded my machine some 2 or 3 months ago, and have reinstalled XP about 6 times or so. I keep getting a bunch of different blue screen stop codes. I tried getting minidumps and full crash dumps, then opening on WinDBG and tried to figure out what the problem is, but I can't find anything consistent. (I've got all kinds of red herrings, like issues with the antivirus, usb camera drivers, ntfs.sys, etc).

So here are the main specs:
CPU: Intel E8400
MB: Gigabyte GA-P35-DS3L
RAM: 2x1GB Patriot
HD: SATA 500 GB Seagate (OS, Main), IDE 300 GB Maxtor (Backup)

Obviously, latest drivers, XP SP3, etc. I've ran MemTest86 for nights in a row, no issues.

So yesterday Windows Live FileShare crashed on me, and the machine wouldn't boot, showing a BSOD with the STOP 0x00000024 error (yet again). I looked up online, and most references pointed to the ntfs.sys file. So, I booted with my Linux SLAX CD, and tried to rename/remove the file and copy it from the WinXP installation disc. SLAX said it couldn't delete the file (I guess renaming in Linux is a copy-then-delete method).

Crap. (I have a suspicion this is related to BioShock's DRM with the SecuROM method, since my hard drive shows crap even on Linux and can't open some folders where SecuROM is installed).

So I rebooted from XP, see if I could get anywhere, and now I got a "NTLDR is missing" before windows even started.

That's really bad.

So I thought: "ok, let's bite the bullet and either run the recovery console, or reinstall WinXP AGAIN."

So I booted with the XP CD, and it showed my 500GB drive as "free space".

Not good. I didn't backup everything!

So I said, ok, maybe the SATA drive is bad, so let's install it to my trusty 300GB IDE drive. I did and then rebooted.

As the first part of the XP installation finished, the machine rebooted and ran a CHKDSK on the D:\ drive (which now is the SATA, the bad one). It found and corrected a bunch of errors (like 15 or so). Then the second part of the XP installation began.

At this point I thought: "Ok, I don't really want to lose my non-backed files, so maybe the SATA drive is better?!?". So i turned off the machine while XP install was waiting on me to begin the second phase.

I rebooted, and chose the SATA drive as the boot drive...

And it works like nothing ever happened.

The corrupt files on the c:\found.000 folder were MP3 files, so I did have a backup of those.

So we'll see what happens next on this saga...

Program Defensively

There's nothing worse than running an optimized build, where the symbols are gone and the locals wrong, because an edge condition happened. And yes, we can play tough and decide that such cases are so rare, we don't put a check for it.

And by Murphy's law, nothing happens the first six months; then you get a crash a couple of days before submitting to your publisher, and it will take the best guys in your team to track it. And very probably, a very simple edge condition will have caused this.

For example:
int * FunctionThatNeverFails( int nSlots, int * pOriginal )
{
int * pNewData = new int[ nSlots ];
memcpy( pNewData, pOriginal, nSlots * sizeof( int ) );
return pNewData;
}

A lot of things can go wrong here. So let's look at the defensive function:

int * FunctionThatNeverFails( int nSlots, int * pOriginal )
{
ASSERT( nSlots > 0 );
ASSERT( pOriginal != NULL );
int * pNewData = new int[ nSlots ];
ASSERT( pNewData != NULL );
memcpy( pNewData, pOriginal, nSlots * sizeof( int ) );
return pNewData;
}

Too much typing? Overkill?
Yes, but it will trap a bunch of things we are sometimes lazy (or just too confident) won't happen at all.

Preprocessor defines

Ok, so say you wrap your assert handler inside this:

#define ASSERT( condition ) if( g_bEnableAsserts) { \ ShowDialogBox( #condition ); Break(); }

Then you use it in your code:

void MyFunction( int n, int m )
{
ASSERT( n >= 0 )
ASSERT( m <>}


Notice the missing semicolon on the first assert? This will compile just fine, since if you expand the macro, it's a valid syntax. However, it makes it not look nice, and may cause confusion.

To fix this:

#define ASSERT( condition ) do{ if( g_bEnableAsserts) { \ ShowDialogBox( #condition ); Break(); } } while( 0 )


It doesn't look as nice, but the compile will now ask you to add the semicolon at the end.

Post & PreIncrement...

Please don't use post-increment. It is just evil.

So use ++i instead of i++.

Why?

Because I've seen a *lot* of code like this:
int nValue = pData[ i++ % 32 ];

When they were clearly trying to do this:
int nValue = pData[ ++i % 32 ];

And actually, it should be even clearer:
++i;
int nIndex = i % 32;
int nValue = pData[ nIndex ];


Remember to code for readability!

(Bug fixed)

Beaten on __restrict

I had some drafts for writing about __restrict, but Elan Ruskin wrote two excellent posts on his Some Assembly Required blog.

I guess you can read about it there :)

Programming is Art...

Despite being told otherwise most of the time, I firmly believe programming is Art.

Why?

It is the process of translating abstract ideas, into concrete sequences of orders for a machine to follow. As such, it involves different trains of thought for each individual, different and unique ways to solve a problem or arrive to a result, much like painting or drawing/sketching. Each person will interpret the idea (problem), and express is based on his/her experience. That is what adds the human experience to it.

Programs are never finished; just like sculptors and painters never feel really finished with their works. As you grow and become more experienced, as your sensibilities around the world change, as your ideasd mature, you find your previous work more childish and immature. You wish to go back and rewrite something you did or modify it to become perfect. However, it is an unattainable perfection. And that is what Art is all about.

"In every passionate pursuit, the pursuit counts more than the object pursued."

DiffMerge

DiffMerge is a wonderful diff tool for files and folders. And it's free! I actually changed my Perforce default tool (p4diff) to use it, since it shows whitespace and per-character differences.

I've heard people recommend WinMerge too, but I think I can't do folders. On some things I'm working on, I need to check if code or assembly files changed between folders...

My only complaint is that it doesn't do diffs on binary files. Shame! It'd be the greatest tool ever if it did show the hex values like on a Hex Editor and the strings, and mark the differences...

STL Allocators

Finally, a clearly explained and concise way to write your own STL allocator... Courtesy of the Visual C++ Team.