Status update 11th March 2012

Looks like storage objects will be making an appearance in a Warzone near you sometime in the future 

See: http://forums.wz2100.net/viewtopic.php?f=35&t=8756&start=150#p97205 for details.

Overview

I'm becoming increasingly convinced that storing JS state in .ini files is the wrong direction to be taking.

I'm going to start by outlining some of the big problems I see, and then I'm going to recommend robust solutions to them.

Problem definition

Currently, the JS API tries to help JS developers with persistence by storing pretty much everything that's in the global scope in savegame files.

The idea is that JS developers shouldn't have to worry about persistence, it should just magically happen.

I have several issues with this approach:

It's unreliable

It's very easy for a scripter to put something on the global scope unintentionally without realising it. And, because the global scope is part of the scope chain for all objects (except null, and possibly undefined too?) they are unlikely to notice because to them everything will seem to work fine.

And, because developers aren't having to think about persistence, it's all to easy to be careless and include functions in global objects which leads to this sort of thing:

statusX=@Invalid()
statusY=@Invalid()
statusTime=@Invalid()

And those errors only tend to surface when an end-user runs in to bugs after saving then loading a game.

Today comes before tomorrow

We can solve today's problems by adding more kludges, but tomorrow will bring new issues that we weren't aware of today.

What I mean by this is that by using .ini files, that are not designed to store Javascript objects, we're going to keep running in to emergent problems.

Scripts are going to get more and more complex. We're already seeing AIs start to use dynamic research paths, I myself am working on an AI that will use completely dynamic objects constructed based on game state and events, etc. There are also fringe cases that mandate the use of eval(), and eval() runs code in the global scope = easy to create gobals without realising it. (Note: Do not solve this by removing eval function, that would be a massive step in wrong direction!)

You're either going to end up making the persistence code increasingly more kludgy over time (which will likely also break backwards save game format compatibility) or you're going to have to get very forceful over what can be persisted in save games (which is defeating the goal of making life easier for JS developers).

It's unreadable

The .ini files aren't designed for storing Javascript objects. They are designed for storing very simple name=value pairs, optionally in [sections].

That's why you end up with cruft like this when trying to store JS objects in them:

mgCyborgStats=@Variant(\0\0\0\t\0\0\0\x2\0\0\0\n\0\0\0$\0\x43\0y\0\x62\0o\0r\0g\0\x43\0h\0\x61\0i\0n\0\x31\0G\0r\0o\0u\0n\0\x64\0\0\0\n\0\0\0\x1c\0\x43\0y\0\x62\0o\0r\0g\0\x43\0h\0\x61\0i\0n\0g\0u\0n), @Variant(\0\0\0\t\0\0\0\x2\0\0\0\n\0\0\0\x16\0\x43\0y\0\x62\0R\0o\0t\0M\0g\0G\0r\0\x64\0\0\0\n\0\0\0\x16\0\x43\0y\0\x62\0o\0r\0g\0R\0o\0t\0M\0G), @Variant(\0\0\0\t\0\0\0\x2\0\0\0\n\0\0\0\x18\0\x43\0y\0\x62\0-\0\x42\0o\0\x64\0-\0L\0\x61\0s\0\x31\0\0\0\n\0\0\0\x1a\0\x43\0y\0\x62\0-\0W\0p\0n\0-\0L\0\x61\0s\0\x65\0r)
rocketCyborgStats=@Variant(\0\0\0\t\0\0\0\x2\0\0\0\n\0\0\0 \0\x43\0y\0\x62\0o\0r\0g\0R\0k\0t\0\x31\0G\0r\0o\0u\0n\0\x64\0\0\0\n\0\0\0\x18\0\x43\0y\0\x62\0o\0r\0g\0R\0o\0\x63\0k\0\x65\0t), @Variant(\0\0\0\t\0\0\0\x2\0\0\0\n\0\0\0\x1c\0\x43\0y\0\x62\0-\0\x42\0o\0\x64\0-\0\x41\0t\0m\0i\0s\0s\0\0\0\n\0\0\0\x1c\0\x43\0y\0\x62\0-\0W\0p\0n\0-\0\x41\0t\0m\0i\0s\0s)

I don't know about you, but I take one look at that and my will to fix a bug is gone. If there's a problem, and considering all contributors have very limited time to work on Warzone, we need something that's much more readable.

And while there might be alternate ways to store JS objects in ini files, they too are prone to failure because ini files weren't designed for this sort of thing!!

It's opaque

When you look at a script, it's not clear what is supposed to be persisted and what isn't.

At the moment, it's not too bad, because scripts are relatively basic and generally only worked on by a single developer.

But as things evolve, scripts are going to get bigger, more complex and be worked on by multiple people.

For example, let's say developer #2 goes in to change something in developer #1's script. They change the values stored in a variable, but don't realise it's getting persisted to the save game files. Two problems arise:

1. They could put functions in there, or other stuff save game format can't handle
2. Any changes at all to a persisted variable will likely break all existing savegames – because they will have the old data structure

This is just one very basic scenario. As scripts become increasingly complex it's going to become more and more painful to deal with a "magically does it for you" save game approach using ini files.

My solution

My proposal is as follows:

1. Add sessionStorage and localStorage objects to the global object
2. Persist their data using JSON format stored in .js files

I'll describe this in greater detail in the "Implementation" section later on this page. But first, let's discuss how the approach will solve problems...

Make it reliable

JSON is the same notation used by JavaScript. Thus, if you can do it in JavaScript, it can be stored in JSON.

We would still want to filter out functions or anything not based on standard classes / primatives. This can be achieved using the .setItem() property of the sessionStorage / localStorage object (see "Implementation" for how).

By detecting undesirable stuff at the point where data is put in to the sessionStorage / localStorage object (I'll just refer to these as the "storage object" from now on), an error can be thrown. This means that:

• Developers will be alerted to problems much sooner - as soon as they put something in to the storage object, rather than after loading a saved game
• Save games will never contain invalid data - because invalid data can't be put in to the storage object
• The .setItem() method on the storage object can provide a more descriptive error message explaining clearly what is wrong = helps dev to fix the problem

The internet has had lots of Today's

...billions of them in fact, considering the number of people that use the internet every day.

Millions of people use the Internet every second. Hundreds of thousands of JS developers write scripts every day. Thousands of developers work on browsers and ECMA-262 implementations, using everything form Java to Perl, every day. And this has been going on for over a decade.

If it works for them, my guess is that it will work for us too.

It's not surprising that an approach used on the internet is going to be much more reliable than a bespoke approach used in Warzone. The internet is tried and tested on a scale that Warzone code will never achieve, simply because the internet is ubiquitous.

So, if things like storage objects and JSON are widely used on the internet, and proven to work over many years by huge numbers of people, I think Warzone can benefit a lot from that. All those little fringe cases that Warzone's scripting future holds... they're all yesteday's from the Internet's perspective, they've already been solved by the internet's approach.

I'm sure that none of the developers working on Warzone source code, or any of the JS devs working on scripts, have set out with a goal of "finding a great new way to persist javascript objects". There are much more productive uses of our time. And in any case, the internet has already come up with a tried and tested way of doing it - why are we trying to reinvent the wheel using, of all things, ini files?!

Make it readable

Look at the code samples I posted in the "Problem Definition" section, and compare to this sort of thing:

{someKey:{"bar":1,":":false,"large pants":"eaten too much",bleh:{"handles nesting":true,"handles arrays":[true,"yes"]}},
anotherKey:{"meh":"fish","[bleh]":false,"large pants":"eaten too much",bleh:{"handles nesting":true,"handles arrays":[true,"yes"]}}}

If you can write Javascript (which I think we can assume is the case for anyone writing javascript ) then you can read JSON. You might prefer to paste it in to a text file and add some line feeds & indentation to make it more readable, but either way, it's a vast improvement over the current .ini format or any proposed alternatives.

JSON is a tried and tested way of doing exactly what it says on the tin: parsing and stringifying JavaScript Object Notation. It's built for that, it's the whole reason JSON exists and it's so widely adopted that I think it's safe to assume it will work as expected (because, in practice, it's used on billions of web page views per day and it does exactly that).

Make it transparent

Using storage objects will make it blatantly clear what data should be stored in the save game file.

A simple search for "sessionStorage" will instantly identify all interaction with persistable or persisted data.

This approach ensures that the intention of the original developer is clearly visible in their source code. There's no magic, no guesswork.

It also has the added benefit of making a developer *think* about what they're persisting. And if there's an error thrown by .setItem() it's much easier for the developer to find the line of code where they did something wrong.

Some strange rebuttals / justifications for the .ini file approach

I know time has been invested in the current implementation. When you invest time in something, you get personally attached to it and don't want to be told that it's wrong. That's just human nature, I'm exactly the same.

Here are some examples of personal attachment to the current .ini approach:

The main reason is these files aren't meant to be edited

Then why are you using .ini files in the first place? Surely everything should be stored as a binary blob that's encrypted instead?

And then you're going to have to spend time writing an editor app so that .gam files can be debugged. How is that going to make warzone any better?

The simple fact of the matter is that the casual gamer will try renaming the .gam file to .zip, and if they can't unzip it they'll give up. After all, the game provides documented cheat system which is much easier to use.

I think it's safe to say that the .gam format successfully prevents casual gamers from hacking save game files. So, all the files within the .gam don't need to be obfuscated - let's make them as readable as possible so we can debug them without having to develop a .gam decompiler!

With custom INI parser we could have things like...

Why bother? Just use JSON - it's tried and tested and known to work.

Adding yet more custom stuff to Warzone is not going to improve code quality, readability or reliability!

We don't want to add more files to save games

(I don't have a link for this one, it's buried in forums somewhere).

There are already loads of files in .gam files. And it doesn't really matter how many files there are. What matters much more is that save games are reliable and easy to debug.

We want to make life easier for the scripter [when it comes to persisting data]

(Again, no link, it's buried in scripting forum somewhere)

Non-transparent code, unreliable persistence, emergent bugs and difficulty debugging persisted data are not my idea of an easy life.

I don't want the JS API to try and magically persist my game state. I want to be able to define, specifically and without any guess work, what will and will not be persisted.

I want to be able to store temporary variables on global without worrying that they'll break the save game files. I don't want to have to keep storing temp variables as properties on functions just to prevent them going in to save games!

If I try persisting something that can't be (or shouldn't be) persisted, I want the JS API to throw an error at the moment I try persisting it. I don't want to have to read through an obfuscated savegame to try and work out if I've persisted something wrong. I don't want problems to surface only after loading  savegame as that makes the dev cycle too long and cumbersome.

Implementation

Implementation should be relatively easy to achieve...

Creating the sessionStorage object

Here's one I made earlier:

Including that file will create a global sessionStorage object that is practically identical to what you'd get in a browser. Alternatively the same sort of thing can be created using C++ code, but IMHO there's no real benefit to doing that.

I say practically identical, but there's two key differences:

• Values can be any data type supported by JSON (in a browser, they'd have to be strings). JSON currently has native support for:
  • Object
  • Array
  • String
  • Boolean
  • Number
  • Null
• I've added toJSON() and fromJSON() methods.

The .setItem() method will, by default, check that the data you're trying to store doesn't contain any unsupported types (eg. Function, Date, etc). If you pass in a nested object or array, it will be examined up to 10 levels deep (easily enough for everything I've seen so far). The checking can be switched off (for extra performance when working with huge objects, or where human knows best) with an optional third param being set to true.

Making sessionStorage available to scripts

My understanding is that the C++ code fires up a JS engine and populates it with the Javascript API (globals, functions, constants, etc). It then effectively "includes" the script file associated with the AI or whatever. That being the case, it's going to be fairly trivial to include sessionStorage.js before the main AI script.

Persisting data in save games

Each instance of sessionStorage running in a game (ie. one for each player) will get it's own .js file in the .gam (save game) file.

Naming convention:

player<playerID>_sessionStorage.js

For example, player #4 would get a file called: player3_sessionStorage.js

The contents of that file are completely derived by calling sessionStorage.toJSON() in the the player's script environment. Nothing else needs to go in that file.

Event triggered before saving

There are some scenarios where a JS dev might prefer to pump data in to sessionStorage immediately before a game is saved, rather than drip-feeding data in throughout the game.

As such, an event would be useful:

eventBeforeGameSave()

This event would be triggered before the sessionStorage.toJSON() method is called, allowing a JS dev to get any additional data in to sessionStorage immediately before the game gets saved.

Loading data

After sessionStorage.js is included in the JS environment, the contents of the playerX_sessionStorage.js file are loaded and then pumped in via sessionStorage.fromJSON(data) (where 'data' is the string representation of the contents of the playerX_sessionStorage.js file).

That's it.

Future development: localStorage

If the sessionStorage idea proposed above gets accepted, then an obvious next-step would be to implement localStorage.

localStorage works in pretty much the same way as sessionStorage with some notable differences:

• It persists across multiple games (whereas sessionStorage is just for a single game)
• It is scoped to a script (whereas sessionStorage is scoped to a player)
• Each time data is added/changed, the associated localStorage objects all need updating (whereas sessionStorage files only get updated when saving a game)

I would assume that we wouldn't want localStorage data being sent across the intertubes during a network game, which is fine - the localStorage for an AI script (eg. NullBot) would be taken from the client machine on which the script is running. If the same script is running on other machines elsewhere, then they will use their own local (client machine) localStorage data file.

In order to enable localStorage, a new JS API function would need exposing, something like:

setLocalStorage(fromPlayer,key,val)

Inside the localStorage.setItem() method, it would do the normal stuff - in other words update its own internal data object. But then it would also call setLocalStorage(me,key,val)

When setLocalStorage is called, the C++ code would do something like this:

// simplified pseudo code
 
function setLocalStorage(fromPlayer,key,val) {
  // work out what script called setLocalStorage (eg. nullbot_1.1.js):
  var playerEngine = getEngineForPlayer(fromPlayer);
  var playerScript = playerEngine.scriptFileName;
  // update all other players where applicable:
  for (var i=0; i<maxPlayers; i++) {
    // no need to update the script that triggered setLocalStorage():
    if (i != fromPlayer) {
      // is this player's engine running same script as the calling script?
      var engine = getEngineForPlayer(i);
      if (engine.scriptFileName == playerScript) {
        // update it's localStorage with the new data
        engine.globalObject.localStorage.setItem(key,val,false,true); // 4th param tells setItem() not to call setLocalStorage()
      }
    }
  }
  // now persist the localStorage data to file on disk
  var jsonData = playerEngine.globalObject.localStorage.toJSON();
  var fileName = "path/to/config/folder/"+playerScript+"_localStorage.js";
  createOrReplaceFile(filename,jsonData);
}

Obviously there will be a load of stuff I'm not aware of in the way things work in the C++ environment, and there's probably several alternate approaches that could be used, but hopefully the pseudo-code above illustrates the macroscopic elements that would be required?

So, what do you think?