My application has thousands of records in one particular table that I need to populate the display with. I was wondering if I can implement paging to speed up the retrieval of those records?

In fact he specifically wanted to know if it was possible to do data paging with the SQLRunner class in my air-sqlite library. The answer is yes it works, without even needing any changes to the library. See below for how to do that.

The easiest way to implement data paging (in other words, getting only a subset of a query’s results at a time) in a SELECT statement is to use the LIMIT..OFFSET clause in the SELECT statement.

In summary, you can put a LIMIT clause at the end of a SELECT statement and the result set will only include the specified number of rows:

SELECT * FROM myTable
LIMIT 3

Then to get the next 3 rows, you add an OFFSET value:

SELECT * FROM myTable
LIMIT 3 OFFSET 3

Fortunately, you can use statement parameters for the LIMIT and OFFSET values (although the docs don’t mention this — shame on me!):

SELECT * FROM myTable
LIMIT :limit OFFSET :offset

Then you just specify values for those parameters and you’re good to go.

I tested this using the SQLRunner class by adding a new set of unit tests and it worked just fine without any changes to the library. (Hooray for unit tests and FlexUnit support in Flash Builder 4 — they made it nice and easy to test this out since I already had the infrastructure in place.)

Here is the SQL statement I used for the unit test using LIMIT..OFFSET with parameters:

SELECT colString,
colInt
FROM main.testTable
LIMIT :limit OFFSET :offset

And here is the line of code that calls the statement (modified slightly for readability). test_result is the result handler function. The parameters object specifies that I want 7 result rows, starting with row number 3 (i.e. skipping 2):

_sqlRunner.execute(SQL, {limit:7, offset:2}, test_result);

As Peter points out in the comments, another approach to do something similar to data paging with AIR is to specify a number of rows to retrieve as the first argument to the SQLStatement.execute() method, then call the SQLStatement.next() method to retrieve additional rows from the same statement. This technique is described in the documentation so I won’t go into more detail about it here except to say that it does have a couple of drawbacks (mentioned in my comment below) that make it less suitable for data paging (but still very useful).

Warning! To add the missing feature I had to introduce a non-backwards-compatible api change. Read the details in the project history.

So, the bug is fixed, the feature is added, the version number is incremented (0.1.1 beta), and the code and SWC are live on Github.

Download the version 0.1.1 SWC

Read about the changes

In conjunction with the release I also added a couple of new examples to the project page, including a “bare bones” code only example for quick starters, and links to an example of using the library in a Robotlegs application (the example app from the 360|Flex training).

Enjoy, and as always feel free to report problems as issues in Github or in the comments on the project page.

For lack of a better name, I call it my “AIR SQLite utility library”

The code currently contains one major piece of functionality (well, two different variations on one bit of functionality), which is a SQL “query runner” library. This is a wrapper for the AIR SQL classes that allows you to run a SQL statement by just passing a few bits of information:

• The text of the SQL statement itself
• An object containing properties with the values for any statement parameters in the SQL
• A Function to call when the operation completes
• A failure Function
• (optionally) a class to use as the data type for the data returned from a SELECT statement

The library does all the work of creating SQLStatement objects and caching prepared queries, as well as pooling SQLConnection instances so you can execute multiple statements simultaneously. It also has a variation that allows you to specify a “batch” of statements to execute, and they are executed in order in a transaction.

I’ve also got an additional utility to add to the library. It’s a “database copy” utility that allows you to create a “deep copy” of a database — all it’s tables, views, etc. — with or without data. The key reason why this is useful is that you can use it to create an encrypted database from an unencrypted database (and vice-versa). It’s written and tested, but I decided to modify the structure slightly before releasing it, so it’s not checked in yet.

I’ve put the details about how it works and why it’s designed that way in the project page. In case you’ve ever wondered how I design apps, I think the examples and this library give some insight into how I actually do my database-driven AIR app development. At least, how I structure the data-access part of my apps.

On (another) personal note, this project is also my first project that I’ve posted to my Github repository (as opposed to projects I’ve forked). It was actually posted-but-not-advertised on Google code for a month or so, but I decided to move to Github because the collaboration and checkin-without-network-connection features are so awesome.

Normally in SQL, including in AIR’s built-in SQLite database engine, you can only add data to one table at a time using an INSERT statement. In some cases, particularly when you’re doing a “bulk loading” operation such as importing data from a text file into a database, it’s convenient to be able to insert data into multiple tables using a single INSERT statement.

For example, suppose you have XML data to import into a database such as the following:

<?xml version="1.0"?>
<employees>
    <employee firstName="Bob" lastName="Smith"
        location="San Francisco" country="USA"/>
    <employee firstName="Harold" lastName="Jones"
        location="San Francisco" country="USA"/>
    <employee firstName="Tom" lastName="Donovan"
        location="Boston" country="USA"/>
    <employee firstName="Mike" lastName="Wilson"
        location="Calgary" country="Canada"/>
    <employee firstName="Steve" lastName="Thomas"
        location="London" country="UK"/>
    <employee firstName="Joe" lastName="Nelson"
        location="London" country="UK"/>
    <employee firstName="Juan" lastName="Varga"
        location="Buenos Aires" country="Argentina"/>
    ...
</employees>

The XML data isn’t normalized, so there is duplicate data between the various records. We will import it into a database with the following (normalized) structure:

Assuming the data is going to be imported as a single user operation, it would be painful to need to prompt the user or throw errors for every duplicate entry.

Using the technique described here, you can use a single SQL statement to add an employee and if necessary any related data including office location and country. (You would still loop over the data and execute one INSERT statement per employee record — but you wouldn’t need to execute three INSERT statements per employee record, or need to check for duplicate office locations and countries for each employee record to be inserted.)

As mentioned above, this technique is probably only appropriate when you’re doing “bulk importing” of data. If you’re just adding a single conceptual record (even if it includes data in multiple tables) you’ll most likely want to use a series of INSERT statements to add the data, so that you can have more precise error checking and handling.

How to do it

In an attempt to “cut to the chase” I’m going to put the necessary code here. For more detailed explanations on how this works and why it’s necessary, see the “details” section below.

This technique for inserting data into multiple tables using a single SQL statement consists of three elements:

1. A view in the database that groups the data to be inserted (from all the tables) into one “table”
2. An INSERT statement to add the data, using the view as the destination “table” in the INSERT statement. This is the INSERT statement that you’ll run from your application while importing the data
3. A trigger defined on the view, which runs when any INSERT statement is executed against the view. This trigger does the actual work of checking for existing data and inserting data into individual tables.

Here’s the code for each part:

A view grouping the data to insert

This combines all the tables that potentially need data inserted into a single “table” for the INSERT statement. You run this statement once to create the view in the database.

CREATE VIEW employees_for_insert AS
SELECT e.firstName,
    e.lastName,
    l.name AS locationName,
    c.name AS countryName
FROM employees e
    INNER JOIN locations l ON e.locationId = l.locationId
    INNER JOIN countries c ON l.countryId = c.countryId

An INSERT statement that “inserts” the data into the view

This code isn’t actually run, but it passes all the data to the database engine for use in the trigger. You execute this SQL statement once for each record to add to the tables.

INSERT INTO employees_for_insert
(
    firstName,
    lastName,
    locationName,
    countryName
)
VALUES
(
    :firstName,
    :lastName,
    :locationName,
    :countryName
)

An INSTEAD OF trigger defined on the view

This is the code that actually runs when the INSERT statement above is executed. You run this code once to create the trigger. Then the database runs the code in the trigger body automatically. If a location or country doesn’t exist it is inserted. However, if they do exist, nothing happens (no duplicate record is created). Then the main employee record is inserted.

CREATE TRIGGER insert_employees_locations_countries
INSTEAD OF INSERT
ON employees_for_insert

BEGIN

INSERT INTO countries (name)
SELECT NEW.countryName
WHERE NOT EXISTS
    (SELECT 1 FROM countries
     WHERE name = NEW.countryName);

INSERT INTO locations (name, countryId)
SELECT NEW.locationName, countries.countryId
FROM countries
WHERE countries.name = NEW.countryName
AND NOT EXISTS
    (SELECT locationId
     FROM locations
     WHERE name = NEW.locationName);

INSERT INTO employees (firstName, lastName, locationId)
SELECT NEW.firstName,
    NEW.lastName,
    locations.locationId
FROM locations
WHERE locations.name = NEW.locationName;

END

Details

The SQL language is designed for working with relational databases, so in a SQL database you usually use multiple tables to represent a single piece of data. That means that in a normal scenario, if you want to add a new record to a table, you may need to add a new row of data to additional tables that the main data is related to.

In this example, we are using a database with the following structure:

This data represents employees in a large company that has multiple office locations, identified by records in the locations table. In fact, this company is an international company that has offices in different countries, including multiple offices in some countries (represented by the countries table.

Each employee is associated with their primary office location by the locationId field in the employees table, and each location is defined as being in a certain country by the countryId column in the locations table.

In order to add a new employee you would generally have to perform several steps:

1. Check whether the country where the employee’s office is located exists in the countries table
2. If not, add it; if so, get its id to create the relationship with the locations table
3. Make sure the office location exists in the locations table
4. If necessary add the location record, and get its id to use in the employee record
5. Add the employee record to the employees table

This is a fairly complicated process because a single SQL INSERT statement can only operate on one table at a time. In a simple case where you are adding a single employee record, this complexity isn’t unbearably difficult. You can execute the series of SQL statements in sequence in a single transaction. If an error occurs, you can break out of the sequence and display an error message or handle the error as desired.

However, if you’re importing a large set of data it’s not convenient to throw errors or display dialogs to the user. Instead, it would be nice to be able to just add any dependent data where appropriate, and insert all the data in one step.

The technique that’s described in this article makes use of database views and a SQLite feature known as INSTEAD OF triggers.

A view is a predefined SELECT statement that’s saved in a database so it can be used in queries as though it was a table. Because it usually includes data from multiple tables, and doesn’t necessarily include all the data from any given table, a view is generally read-only — you can use a SELECT statement to retrieve data from a view, but you can’t use an INSERT, UPDATE, or DELETE statement to modify the view data.

However, in SQLite (and consequently in AIR) you can define a special type of trigger that’s associated with a view known as an INSTEAD OF trigger. (A trigger is a set of SQL code that’s associated with a table. The code is executed when a data manipulation operation is performed on that table.) When a SQL statement attempts to perform an INSERT, DELETE, or UPDATE operation on the view that has an INSTEAD OF trigger defined for that particular operation, the trigger is executed instead of the specified operation. You can only define INSTEAD OF triggers on views (it wouldn’t make sense to use one for a table). For more information about triggers, see the Adobe AIR SQL reference documentation for the CREATE TRIGGER statement.

In this example, a view is defined that includes all the data in all the tables that potentially need data inserted. When the INSERT statement is executed the database runs the trigger instead. The trigger contains code that checks whether related data in the locations and countries tables already exists, and adds it if necessary. Then it adds the employee record with the related key values.

Adobe AIR data privacy and security slides, notes, and links (1 MB .zip)

As a side note for those who actually attended the presentation, in retrospect I think I over-emphasized the security concerns and didn’t emphasize enough that there are plenty of use cases for which AIR is definitely secure — especially in the case where you need to keep the user’s private data secure. Hopefully the notes that accompany the slides help to clarify this somewhat.

I also used and referred to a number of resources in my presentation, which are listed below. The download .zip with the slides also includes an html page with all these links.

Introduction

• "Maintaining security with Adobe AIR" by Ethan Malasky and Peleus Uhley (Adobe MAX 2008)

Background

• "AIR security" (Adobe AIR documentation)
• "Why Adobe AIR Doesn’t (Yet) Include the Feature You Most Want" by Oliver Goldman
• "Understanding the Flash Player Security Model" by Deneb Meketa (Adobe MAX 2008)
• "Considerations for using encryption with a database" (Adobe AIR documentation)
• "Using digital rights management" (Adobe AIR documentation)
• Operating system security (user and "admin" rights)
  • "What are Adminstrative Rights, Anyway?" by Oliver Goldman
  • "Does Installing an AIR Application Require Admin Rights?" by Oliver Goldman
• Source code visibility
  • Action Script Viewer (ASV)
  • "Ethical SWF Decompiling" by Lee Brimelow
  • Nitro-LM
• Encryption: ActionScript crypto libraries:
  • as3crypto
  • OpenSSL (partially) cross-compiled to ActionScript using Alchemy

AIR application installation

• Sign your app with a trusted cert
  • "Code Signing in Adobe AIR" by Oliver Goldman
  • "Digitally signing Adobe AIR applications" by Todd Prekaski
  • Promotion: get a free signing certificate (while supplies last) by submitting your app to the Adobe AIR marketplace
• Plan for updates
  • "Building AIR applications that can be easily updated" by David Deraedt
  • "Using the Adobe AIR update framework" Quick Start article by Jeff Swartz (Adobe AIR documentation)

Modular applications

• Sandbox bridge
  • "Scripting between content in different domains" (Adobe AIR documentation)
• XML signature validation
  • "Creating and validating XML signatures" by Joe Ward
  • "Using the XML signature validation classes" (Adobe AIR documentation)
  • "flash.security package" reference (Adobe AIR documentation)

Local shared objects

[No links]

Encrypted Local Store

• "Storing encrypted data" (Adobe AIR documentation)
• "EncryptedLocalStore class" reference (Adobe AIR documentation)

Local files

[No links]

Local SQL database (SQLite)

• SQL injection attack
  • "Using parameters in statements" (Adobe AIR documentation)
  • "SQLStatement.parameters property" reference (Adobe AIR documentation)
• Encrypted database
  • "Using encryption with SQL databases" (Adobe AIR documentation)
  • "Using the EncryptionKeyGenerator class to obtain a secure encryption key" (Adobe AIR documentation)
  • as3corelib project (includes the EncryptionKeyGenerator class and hashing algorithms

A couple of years ago, before AIR beta 1 (“Apollo”), I wrote a tool for testing SQL statements in AIR. Later (I can’t remember exactly when) I released it under the name “Doppler.” I had plans to make it a full-fledged database management tool, but those plans never really panned out. I’ve told this story lots of times now, about how I now use (and even help develop) David Deraedt’s “Lita” SQLite admin tool.

However, “Doppler” in its current form served a different need than Lita — Lita is great for creating and managing database structure (you’ll never have to write a CREATE TABLE statement again), but it isn’t as strong for testing individual SQL statements. Over the past several months I’ve been doing some “serious” app development, working on an AIR app that makes heavy use of SQLite. I found my little tool to be quite handy, and also found and implemented a number of small improvements that make it much more useful.

Since my original intentions were never going to come to fruition, and I had new future plans for the app, I decided a new name was in order. Since it is for running SQL statements, “run” was an easy and obvious choice. So now, (several weeks after posting it), allow me to officially introduce you to the Run! AIR SQL query authoring and testing tool — the next generation of Doppler.

If you look at the full feature list for the latest release, you’ll see that I’ve added several types of changes. A few of the ones that I’ve found most useful in my day-to-day work are:

• Auto-update and badge install — no more downloading the .air file (admittedly not a big deal) and no more uninstall and reinstall each time there’s a new version (a big deal)
• “Recent SQL files” and “Recent databases” menu items
• Opening and saving SQL files (plus keyboard shortcuts)
• “Row count” and “execution time” displayed for SELECT statements
• Line numbers and auto-indent in the SQL panel

And yes, I did say “day-to-day” work. On some days, Run! has been my primary IDE, as I’m writing and testing and rewriting large SQL statements. It’s been kind of strange and kind of fun building a developer tool and using it as much as I use Run!.

I’ve already gotten a bug report and I’m working on features for the next big release, so if you have thoughts or comments please comment on the project page!

As David pointed out, the fact that I’m an Adobe employee and am participating in the project doesn’t mean it’s now an official Adobe product (for good or bad). I’m doing this 100% on my own time. And my role is still pretty small — David is certainly the lead, main, primary, controlling, etc. author. I just file bugs as I find them, add some comments about feature requests, and fix issues when I know how to.

In case you’re curious, the back story is really pretty straightforward. I started writing my own version of an AIR SQLite admin tool back before AIR beta 1, but never had time to take it beyond a “query runner” tool. Late in 2008 I discovered Lita, and once I started using it I realized that 1) it is implemented in a similar way to many of the ideas I had, and 2) It’s already got a big head start in features, so there’s not much point in me trying to “compete” or anything like that, especially for something that I wasn’t planning to make money from.

After using it for a while I discovered a few bugs, and decided to email David about them. (He and I had communicated a bit previously, about the encrypted database functionality added in Adobe AIR 1.5 and how to integrate it into Lita.) I think I probably mentioned in that message that I would be willing/interested to fix issues myself as I have time. Fortunately David was very kind and accepted my offer. So, as I said, now when I find issues I get to fix them myself, which is nice because I can fix them quickly, but also adds some responsibility since now the burden is on me to make those changes myself =)

As a side note, I really want to point out that David is a really great developer — something I appreciate greatly as we work in the same codebase. It is a big testament to his architectural and coding skills that I was able to dive right in and fix four bugs/feature requests in a very short time (literally a matter of minutes after first looking at the code). I’ve learned a lot just from seeing his code, and now I’m anxious to read his “Flex Architecture Fundamentals” series to learn more about the thinking behind the great code.

As with previous versions, if you’ve been using the app you’ll need to uninstall it before installing the new version. Someday I’ll add updating support, but I’m not going to promise anything real soon.

Along with this release, I’ve also made a change to how I describe the tool, and to my future intentions for it. I’ve always had it in my mind to make this a full-fledged database admin tool, similar to the tools that come with SQL Server or other enterprise databases. However, time has obviously not allowed me to do that, and in the mean time other tools have been released by other developers. I’ve found one, David Deraedt’s “Lita” SQLite admin tool that is sufficiently mature that I use it in my day-to-day work now and it definitely beats doing things by hand! There are still improvements to be made and features to be added, but when I’ve reported bugs and feature requests he’s been quick to respond and release updates.

So while I’m sure nobody’s been holding their breath waiting for me to finish the “admin tool” portion of my app, I just wanted to clarify the new direction I’m taking it — or rather, the fact that I’m not planning to take it in as many new directions! (Hence the change in title for the project from “AIR SQL admin tool” to “AIR SQL query testing tool.”)

That doesn’t mean I’m done developing this tool by any means. In past jobs where I did heavy database development, and in a project I’m currently working on that involves heavy database development, I find it very useful to have two different kinds of database tools — one for creating and managing database objects and structure, and another for testing queries. While Lita does in fact have a tab for testing queries, I personally find Doppler to be a bit (not a lot, but a bit =) more developed in that specific area. On the other hand, Lita certainly does a lot in the db management space that Doppler doesn’t do. So I find the tools very complementary in terms of my actual development work.

As always, I welcome feedback, questions, thoughts, etc. And thanks to everyone who’s already reported bugs and offered suggestions!

As you may have heard among all the news that’s coming out from MAX right now, Adobe AIR 1.5 has just been released. Most of the features of AIR 1.5 are features that were introduced with Flash Player 10. However, one new feature in AIR 1.5 that helps address the easy-to-read database issue is AIR 1.5’s new support for AES encrypted databases.

I’ve had the interesting and at times complex job of writing the documentation for this new feature. The bulk of the new documentation is in a new section “Using encryption with SQL databases” in the SQL database chapter of the manual “Developing Adobe AIR Applications”.

It’s pretty straightforward to use an encrypted database. You have to create the database as an encrypted db. To create or open it, you call the SQLConnection class’s open() or openAsync() methods, just as you normally do, and there’s a new parameter where you provide a 16-byte ByteArray encryption key for the database. That’s all there is to it.

The SQLConnection class also has a new reencrypt() method, that you use to change the encryption key of an already encrypted database.

If you want to see some code examples, check out the documentation listed above, or see these quick start articles on the Adobe developer center:

• Working with the encrypted local SQLite database (Flex)
• Working with the encrypted local SQLite database (Flash)
• Working with the encrypted local SQLite database (HTML/JavaScript)

There are a couple of parts of the new documentation that I think are particularly interesting (although I’m obviously biased since I wrote it all):

• Considerations for using encryption with a database talks about some of the various reasons you may want to use an encrypted database, and some of the differences in how you might architect your application to account for the desired level of security and privacy.

• Example: Generating and using an encryption key goes into great depth into one technique for creating an encryption key for your database. You have to provide a ByteArray encryption key to encrypt your database. How you come up with that encryption key can have a big impact on the actual security of your app data.

  This was definitely the most involved example I’ve written for the documentation. the techniques it uses were specified to me by security engineers at Adobe. The documentation goes into lots of detail describing how these techniques are used and why. Both the code and the text went through a series of reviews by engineers and security experts.

  I ended up writing the example as a simple UI, plus a class that anyone can just pull into their own app (rather than having to pull apart the example and turn it into something you can use. I’m also excited to share that Mike Chambers and Christian Cantrell decided that this “encryption key generator” class is useful enough that it’s now included in the open-source ActionScript 3.0 core library (as3corelib) project.

Now that AIR 1.5 is out the door, I’ve updated my AIR database admin tool to support encrypted databases (when you try to open an encrypted database it prompts you for an encryption key, which you enter as a Base-64 string). I already mentioned that I wrote the encryption key generator class that’s now in as3corelib. I’ve also got at least one more new encrypted database-related open-source project that I’m about to release…but I’ll wait until the MAX “firehose” dies down a bit before I write more about that one. =)

As I mentioned briefly before, this week I presented at the 360|Flex San Jose (August 2008) conference. My presentation was titled “Adobe AIR SQLite: An optimization conversation.” As I mentioned in the presentation, the term “optimization” could mean a few different things — for example, optimization meaning improving performance, or optimization meaning improving developer productivity. While my presentation focused mostly on the first type of optimization, I included suggestions for tools, libraries, and strategies that fall in the “developer productivity” type of optimization as well.

Anyway, as always I’m happy to make my presentation materials available. Here are the slides (with some notes) from my presentation:

“Adobe AIR SQLite: An optimization conversation” slides (PDF in .zip - 504kb)

I don’t really have any specific code examples, apart from what’s in the slides, so there’s no “source code” download. However, I did link to a lot of external tools and resources (including a few of my own). To save you the trouble of digging into the PDF, here are the links:

Tools

• Cristophe Coenraets’ “SQLite Admin”
• My tool for testing queries
• “Lita” by David Deraedt (I learned about this one after the presentation, so it’s not discussed in my slides/video, but I thought it’d be worth mentioning anyway.)

Application architecture/patterns/libraries

• SQLite MXML wrapper classes (Peter Elst)
• Data access layer (Brandon Ellis)
• asqlib SQL statement generator (Miran Loncaric)
• “Command” classes (me) as demonstrated by my AddressBook sample application
• AIR ActiveRecord source and blog post explaining its usage (Jacob Wright)
• AIR SQLite Cairngorm services (Eric Feminella)
• Connection and statement pools, mentioned (with source code) in the article User experience considerations with SQLite operations (Daniel Rinehart)

Finally, as you may have heard, Adobe sponsored the recording of every presentation at 360|Flex, and they’re all going to be made available free of charge via a channel in Adobe Media Player. They’re rolling them out in phases, and mine isn’t available yet. When it is, I’ll update this post with the video as well. Update: the video is now available on Ted Patrick’s blog as well as in Adobe Media Player.

In the mean time, 360|Flex was full of awesome presentations. I wasn’t able to get to all the ones I wanted to see, due to conflicts and me trying to finish up preparation for my presentation. So I’m going to be spending some time watching many of those videos as well. If you’d like to see the videos, Ted Patrick has posted instructions on his blog:

How to view 360|Flex San Jose 8/08 session videos in Adobe Media Player

(via Twitter from Peter himself)

For some reason, the final time that I saved this post before publishing it, my changes didn’t get saved. So if you saw this soon after it was posted, and wondered why it didn’t really talk about SQL changes, and why it had some messed up headings…that’s why. Sadly I’m sure I can’t recreate the text as well as I wrote it the first time. But here’s my best effort…

First of all, allow me to clear something up. I am aware that Adobe AIR 1.0 shipped almost exactly a month ago, so this shouldn’t be news to anyone who’s actually interested in it. I intentionally held off on publishing my thoughts (and a description of changes since beta 3). As a rule, I never post anything in the week or so following a big product announcement — there are so many “me too” posts about the product, and so little actual different thoughts or original ideas, that I don’t want to clog up the aggregaters by adding to the hubbub (the “MXNA firehose” as I’ve heard JD describe it).

That’s not a complete excuse, of course. The truth is also that I’ve just gotten busy with a lot of post-launch cleanup activities, and of course moving ahead on future projects (Flash Player 10 and AIR 2.0), so I’ve neglected this post for longer than it deserved.

And now, lest I postpone it any longer, here are a few bits of news and opinion from my corner of the Flex/AIR world:

• Local SQL database changes
• What does “finished” really mean?
• Go right to the source

Local SQL database feature changes

Wait a minute — didn’t I say a couple of months ago that as of beta 3 AIR is API frozen? Indeed that was the intention, and with one small exception it’s true. And that was the first thing that came to my mind when I read the notification (shortly after beta 3 was locked down) that some changes to the SQL functionality had been checked in to source control. But I was correct before — the API hasn’t changed — just one particular behavior has.

The specific change is with how the runtime treats column data types in database tables. In betas 1-3, AIR was consistent with the “default” SQLite behavior:

the type of a value is associated with the value itself, not with the column or variable in which the value is stored…. All other SQL databases engines that we are aware of use the more restrictive system of static typing where the type is associated with the container, not the value.

In order to maximize compatibility between SQLite and other database engines, SQLite support [sic] the concept of “type affinity” on columns. The type affinity of a column is the recommended type for data stored in that column. The key here is that the type is recommended, not required. Any column can still store any type of data, in theory. It is just that some columns, given the choice, will prefer to use one storage class over another. The preferred storage class for a column is called its “affinity”.

So under the “affinity” system, if in your CREATE TABLE statement you specify that a column stores INTEGER data, and you write an INSERT statement that stores a String instance in that column, the runtime allows it. (I initially wrote “it works just fine” but that may not be true — since your application may be assuming that values retrieved from that column are always integers.) When I first read about this behavior when we were reviewing the spec for the feature, I recognized that it was different than what I’d done before, but I figured it was okay — I’d just have to be extra careful when inserting values into the database, and enforce the data types myself by convention even if the runtime didn’t enforce them in practice.

However, the feedback we got from developers was that this doesn’t really match what they’re used to from typical web app database code (which likely uses databases such as Oracle, SQL Server, or MySQL). As a result, in the final release, the declared data types of database columns are more strictly enforced. If you declare a data type for a column, then attempt to store a value that isn’t an instance of that type, the database will try to convert the value to the appropriate type (for example, turning a number into a string, or parsing a string to see if its text is actually a number). If the value can be converted, then the converted value is stored; otherwise, the runtime throws an error. (For more details, see the column affinity description in the documentation.)

Personally I’m pretty happy with this solution. It’s still a bit more flexible than the databases I mentioned before — usually they won’t even try to convert a value, and will just choke if the data type doesn’t match. But I think that the way it works in AIR 1.0 is consistent philosophically (and actually) with the way data types work in ActionScript 3.0 where values can be strongly typed or weakly typed at compile time, but either way at runtime the type must match.

So, what do you do if you were one of the people with more SQLite experience than other db experience, of who otherwise preferred the old behavior? Well, there is still a workaround. If you don’t declare any data type for your columns, you can store data of any type without causing errors. This isn’t the same as the previous behavior, but based on some anecdotal observations I’d guess that developers who intentionally mix data types in columns are less likely to declare a column data type, in any case.

Also, as a side note, I decided to keep the term “column affinity” in the documentation even though the behavior changed. I did this for a couple of reasons: 1) Developers who’d been reading the docs were probably familiar with the term now, and 2) the data types still aren’t strictly enforced, in the sense that the runtime attempts to convert values before storing them (just as a default SQLite implementation does). For example, just about anything can be stored in a TEXT column without an error occurring. So in that sense, the data type specified in a CREATE TABLE statement is still somewhat of an “affinity” rather than a traditional strictly enforced data type.

What does “finished” really mean?

Not too many years ago, my definition of a “release version” of software was different than it is now. I knew that any software, no matter how much it is tested, is bound to have some bugs in it — it’s just not possible to ship software of any notable size without making a mistake somewhere. However, I had enough faith in big software companies that I assumed that when a company declares a product to be a “release” or “shipping” version, that it at least doesn’t have any bugs that they know of.

I’m not sure when I realized that this view is far from reality. I suppose it was probably with the first private beta I was involved with, though maybe it was before that.

In any case, I hope I’m not shocking anyone’s sensibilities when I point out the reality, that even though Flex 3 and AIR 1.0 have shipped, they do have bugs — and Adobe even knows it! The recent openness of Flex, in particular the introduction of the public Flex bugbase, makes this reality much more transparent. A quick search on the Flex SDK project, looking only at bugs (as opposed to feature requests), shows 309 “open”, “in progress”, “community”, or “new” issues. And that’s not even taking into account “deferred” bugs, such as SDK-14339 “WindowedApplication doesn’t update/show menu when it changes during runtime”.

While Adobe AIR doesn’t have a public bugbase, that doesn’t mean it’s any more free of bugs. Try calling the NativeMenu.display() method on a PowerPC Mac running OS X 10.4 and you’ll see what I mean (hint: the menu appears at the mouse cursor rather than at the specified coordinates).

So what does “release quality” software really mean? Well, it means that it doesn’t have any bugs that are “showstopper” issues, or that are “really bad.” Yes, I’m intentionally using non-specific terms here. The reality of life is that if Adobe (or any software company) waited to release software until all known bugs are fixed, the software would never get released. So instead, they use a process that’s sort of like this (I’m generalizing and also stating this based on my own observations, not on any review of actual process or guidelines within Adobe):

1. The decision makers figure out a date by which the software should be done. (I’m not talking about at the beginning of a product cycle — I’m starting somewhere in the middle.)
2. Throughout the process, bugs are reported. Any time a bug is reported, it is investigated and a decision is made as to whether resources can/should be allocated to fixing it. The criteria for this decision are generally based on how severe the bug is (whether it crashes the software/computer, causes data loss, is a usability issue, etc); how hard it is to fix; how much time it will take to fix; how likely it is that someone will run into the bug; whether or not there’s a workaround; etc.
3. As the product moves closer to the release date, the threshold for whether a bug will be fixed goes up — the closer the software is to the release date, the more severe and less difficult to fix a bug must be in order for it to get fixed.

This can be frustrating for people who file bugs, like me. Often, due to workload and general stability issues, it’s not practical for me to start writing the documentation for a feature until it’s fairly stable and close to finished. But that means that when I do find bugs, there’s a good chance it’s too late in the process for the bug to get fixed in time for the product release.

Ah, tradeoffs.

Go right to the source

Now that Flex 3 has shipped, the Flex 3 source code is available in the Flex Subversion (SVN) repository. I’ve actually been accessing it there for a while — internally we made the switch from Perforce to Subversion around the first of the year. I was pretty worried at the time, because I had never used SVN before. The majority of the Flex 3 work was already done at that point — they were just in final bug fixing mode. We still had about a week before our content had to be locked down, so I was working furiously to try and finish up some work I was doing on the Flex AIR components. I wasn’t too excited about the prospect of switching source control systems in the middle of that crunch. Fortunately, SVN turned out to be pretty straightforward. (In case you’re wondering, I use TortoiseSVN). It’s actually been good for me to learn SVN at work — it made it easier for me to move some of my code projects over to Google Code (a few that are already there, and a few more that I’ve started but not posted on my projects page yet). So overall I give SVN thumbs up, and I’m happy that the Flex source code is available that way.

If you’ve been following my posts on AIR and the SQL database functionality, you may have noticed a trend: every time there’s a new beta, I tend to go silent for a few weeks beforehand as I try to get things finished up with the documentation. Well, here we went again — another period of silence, another AIR beta.

The marketing team kept things pretty low-key with announcing this new beta, especially compared to previous rounds. I’m not sure of all the reasons for that, but there are a few that are important and helpful for you to know if you’re developing AIR applications:

• AIR is now API-frozen — as of this beta, there shouldn’t be any changes to the AIR apis. In fact, the goal was to have as few API changes as possible between AIR beta 2 and AIR beta 3, as things get closer to being locked down. Naturally, if we find a big security issue that requires us to change an API, we’re going to consider it — but aside from that, things are now the way they will be for AIR 1.0 (phew — it’s been tiring for me to update apps every beta, and I’m sure it has been for you, too).
• Along with the new AIR beta, Adobe also released a beta of a new product code-named “BlazeDS”. This very cool offering is a free, open-source, slimmed down LiveCycle Data Services — basically it includes the RPC Components (“Flash Remoting”) functionality as well as the Messaging functionality. In addition, we’re releasing an official specification for the AMF format. While the format was (for the most part) reverse-engineered long ago, I think it’s great to have something official for all the other open-source offerings like AMFPHP, Fluorine, etc. to work from.
• Along with beta 3, Adobe created the “Adobe AIR Marketplace” where developers can post apps for people to download and try out — basically an extension of the
• Flex 3 beta 3 is officially the last Flex 3 beta before Flex 3/AIR 1.0 (second paragraph, also in the FAQ PDF). ‘nuff said =)

Of course, following the trend of my going silent before each beta, you can expect that to be magnified as we continue leading up to AIR 1.0. There have been a few things each beta that I wanted to do but had to put off until later — but of course, this time there isn’t any “until later” — so it’s now or never!

Talking specifically about the local SQL database functionality, there were only a few changes this beta, but they’re pretty significant and maybe even a bit sneaky so you’ll want to pay attention to them:

• Specifying synchronous and asynchronous execution: Synchronous database execution was added to SQLConnection in beta 2. At that point, to specify that a SQLConnection’s operations were synchronous rather than asynchronous, you had to specify a true parameter in the SQLConnection() constructor. Now, however, the SQLConnection() constructor doesn’t accept any parameters. Instead, you specify the execution mode when you open the connection to the main database. To specify that you want to use synchronous execution, you call the SQLConnection.open() method; to specify async execution, you call the new SQLConnection.openAsync() method.

  The most important thing to note surrounding this change is this: if you’re migrating an app from beta 2 to beta 3, you may be changing the behavior of your app without even realizing it! If you want a connection to be synchronous, you’ll know you have to change things because the code won’t compile (SQLConnection() no longer accepts a parameter). However, if you want you connection to be asynchronous, then if you don’t change anything the code will still work but it will actually use synchronous rather than asynchronous execution. If you have a long-running query and it’s freezing the UI now, that’s probably the reason — the exact code you used previously to specify asynchronous execution (no SQLConnection() parameter, and open() to connect the db) now is the way to specify synchronous execution. If your code doesn’t cause any errors, you might not even notice immediately because the SQLConnection and SQLStatement events are still triggered even in synchronous execution — so your “success” event listeners will still get called (but error listeners won’t — the operations will throw exceptions instead).

• There were a couple of changes to the SQLError class and related classes: The SQLError.code property was removed (because it already inherited errorID from Error) and the details property was added, providing a more detailed error message in certain cases. Along with this, the SQLErrorCode class was removed entirely (tragically — I spent a fair amount of time hammering out those error descriptions with Jason, the engineer for the feature).

I think that’s it, but please let me know if I’ve forgotten another change!

In the mean time, In addition to the SQL database stuff I’ve been doing more with another part of AIR that I’ve previously been working on as I’ve had time — the AIR-related Flex components. This release includes a new one, the FlexNativeMenu component, which is pretty slick. Watch for more on this in the (hopefully near) future!

• Synchronous database operations
• Schema API (database instrospection)
• Additional data types
• Other new documentation
• Bug fixes

Synchronous database operations

From the various public and internal feedback forums I’ve seen (discussion lists, mailing lists, etc.) this is definitely the most-requested feature from beta 1 (at least related to the embedded database). It’s in there now — when you instantiate a SQLConnection object, just add true as a constructor argument and all the operations (including SQL statements) that are executed through that SQLConnection will run synchronously.

The good news is it’s really very similar to the way things worked in beta 1, so if you’ve already been doing things asynchronously you won’t have to do much to get going with synchronous db operations. Note that a lot of people have referred to this as a “synchronous API” but in fact there isn’t a separate synchronous API — it’s the same API, with the exception of the single SQLConnection() argument.

To learn more, here’s the place where you should probably start: Developing AIR Applications > Working with files and data > Working with local SQL databases > Using synchronous and asynchronous database operations (Flex | HTML/JavaScript | Flash)

Schema API (database instrospection)

One limitation that developers pointed out in beta 1 was that there wasn’t any built-in way to get information about the structure of a database, tables, etc. In beta 2, there’s a new set of classes that can be used to get at that information. There isn’t any information in the developers guide about this yet, but in this case you can figure it out by looking in the API reference. The best place to start is probably with the SQLConnection.loadSchema() method (Flex | HTML/JavaScript | Flash).

There are also several new classes that are used for the various types of schema information you can get:

• SQLTableSchema (Flex | HTML/JavaScript | Flash)
• SQLViewSchema (Flex | HTML/JavaScript | Flash)
• SQLIndexSchema (Flex | HTML/JavaScript | Flash)
• SQLTriggerSchema (Flex | HTML/JavaScript | Flash)
• SQLColumnSchema (Flex | HTML/JavaScript | Flash)

Additional data types

Several ActionScript/JavaScript data types can now be used as column data types in the CREATE TABLE statements. When you specify one of these data types for your column, you can pass an instance of that type into an SQL statement (using a parameter) and it will property store and retrieve the data as the ActionScript/JavaScript data type. The new supported data types are:

• Boolean
• Date
• XML

More information about these data types can be found in the developers guide section “Working with database data types” (Flex | HTML/JavaScript | Flash) and in the language reference appendix “SQL support in local databases” (specifically the “data type support” section). (Note that the link goes to the Flex language reference. I couldn’t find that appendix in the Flash or HTML/JS versions, although the content is identical for all of them since it’s really just talking about SQL.)

Other new documentation

Although I did take several weeks between beta 1 and beta 2 to move from Indiana to the San Francisco bay area, I did get some additional documentation written. If I remember right, these are the sections that are new for beta 2 (other than the ones I’ve already mentioned above):

• Using parameters in statements (Flex | HTML/JavaScript | Flash)
• Strategies for working with SQL databases (Flex | HTML/JavaScript | Flash) - new sections on improving performance and best practices
• … and more! (improvements and corrections in various other sections)

Bug fixes

Need I say more? =)

(In seriousness, lots of bugs were fixed, so if there’s an issue you were running into, try it out in beta 2 and let us know if it’s still not working.)

I admit, my JavaScript is much rustier than my ActionScript/Flex, so I’m glad to see these sorts of things — it helps me to see what patterns and approaches work for the larger-than-you-might-imagine JavaScript AIR developer audience.

Update: added links, so that you can actually find the article (Doh!)