Data persistence choices

In roughing out the early data framework for Daily Value, one of the early considerations was data persistence – particularly user created data.  The app’s requirements included supporting user entered food data that could be treated “on par” with shipped food data.

Looking around at what the iPhone SDK supported, a couple choices quickly presented themselves: archiving, property lists, and SQLite.  While the temptation was to leap right into a relational database solution, some consideration was made for the other choices and in truth, Daily Value employs all of these techniques for various pieces of data.

There’s a nice write up in “Beginning iPhone Development” by Dave Mark and Jeff LaMarche so I won’t dive too deeply into it except to say property lists and archiving were discarded.  Although writing data in a binary format with archiving had some appeal for data obfuscation purposes, it would have also meant writing a bunch of “database-like” functionality that didn’t seem a good technical investment in time or resources.  So, SQLite was what I went with.

SQLite offers a lot of upside:

  • The API is written in C and is easy to navigate.
  • Storage and manipulation of the data was offloaded to the database framework.
  • There are a number of free tools that can be used to browse and quickly inspect the data (ex. SQLite Manager for FireFox)
  • Creation can be automated
  • Good performance on complex queries

The biggest drawback is an apparent lack of authentication or way to protect the data placed into the database.  Since our data was publicly available, this wasn’t much of an issue but it might be for some other apps.

The initial design called for all the data – both shipped and user generated – to be stored in a single database file.  The thought being that it’d be easier to code queries against and manage.  User data would contain some differentiator (either a column identifier or field) to denote it as “editable” (you don’t want folks mucking about with your data).

Apps that use persistent data, or write back cannot do so to files stored within their application bundle or else it breaks the code signing and things stop working.  So, when the app initially loads, the source data is copied to the Documents directory and opened from there.  Of course, the app checks first to see if the file already exists so it doesn’t overwrite an existing file.

Test rigs built for the simulator seemed to indicate that this was viable, but putting the app on the actual device proved otherwise.  Apple requires that apps fully launch quickly after no more than 5 seconds.  The first database prototypes were nearly 4 MB.  The bulk initial copy caused startup to be unacceptable taking over 10 seconds!

Trimming the database helped.  With a little bit of effort in making tables more relational and dropping columns that weren’t strictly required, the size got squeezed down to just under 2 MB.  Even so, the copy took too long – so what to do?

After digging about through the SQLite documentation, I came across the SQL “attach” command.  One neat feature in SQLite is that it supports the notion of opening multiple databases and treating them (generally) as a single entity.  Care was taken to create a “template” user database with pre-created tables that were different than the app supplied database.  While not strictly necessary since SQLite will allow you to have duplicate table names, it seemed prudent and an easy “design time” step.

When the app launches, if it doesn’t already exist, the “empty” user database (all 12 kb of it) is copied from the main bundle to the Documents directory and opened.  Then, the app “attaches” to the large bundle database with the restriction that none of the bundled data can be modified.  Startup times were acceptable and the additional complexity was very minimal due to the SQLite abstraction support.

Here’s a bit of code to get the job done – assuming that dbHandle is a member variable of your data class.

- (void)copyDataIfNeeded
   NSFileManager *fm = [NSFileManager defaultManager];
   NSArray *docPath = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
   NSString *docDir = [docPath objectAtIndex:0];
   NSString *writePath = [docDir stringByAppendingPathComponent:@”user.db”];

   BOOL success = [fm fileExistsAtPath:writePath];
   if (success)

   NSError *error;
   NSString *dbPath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent:”user.db”];
   success = [fm copyItemAtPath:dbPath toPath:writePath error:&error];
   if (!success)
      NSAssert1(0, @”failed to create writable database with msg ‘%@’”, [error localizedDescription]);

- (void)attachDatabase
   [self copyDataIfNeeded];

   NSArray *path = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
   NSString *docDir = [path objectAtIndex:0];
   NSString *openPath = [docDir stringByAppendingPathComponent:”user.db”];

   if (sqlite3_open([openPath UTF8String], &dbHandle) == SQLITE_OK)
      NSString *sysDBPath = [[[NSBundle mainBundle] resourcePath] stringByAppendingPathComponent:”main.db”];
      NSString *sqlString = [NSString stringWithFormat:@”ATTACH DATABASE ‘%@’ AS main”, sysDBPath];

      const char *sql = [sqlString UTF8String];
      sqlite3_stmt *statement;
      int success = 0;

      success = sqlite3_prepare_v2(dbHandle, sql, -1, &statement, NULL);
      NSAssert1(success == SQLITE_OK, @”Failed to prepare ATTACH with msg ‘%s’”, sqlite3_errmsg(dbHandle));

      success = sqlite3_step(statement);
      NSAssert1(success == SQLITE_DONE, @”Failed to execute ATTACH with msg ‘%s’”, sqlite3_errmsg(dbHandle));

       NSAssert1(0, @”Failed to open database with msg ‘%@’”, sqlite3_errmsg(dbHandle));

Unexpected additional time for review…

Not really the first (well, second) entry I was thinking of making – but sometimes events aren’t as one would expect.

Daily Value has been in development since April ’09 and was approved for sale in Aug ’09 (see the blog or main page for details) and I’ve been working on an update for the past few weeks.  Not only are there some neat new features planned, but a number of “quality issues” were discovered after our first binary came out.  The first submission went very smoothly – as I recall it took about 6 days for the app to go “live” in the stores.

Obviously, it’s very disappointing that problems weren’t discovered during the testing and stabilization phase and even worse reached our customers.  Not to rationalize too much, but it’s challenging to fully regress a product and anticipate a nearly infinite range of error states and conditions.  That said, I’ve started adopting unit tests to help cover corner cases and prevent functional regression or re-introduction of bugs.  Unit testing is a broad topic worthy of a longer and dedicated post or two – but suffice to say, I’ve found and fixed a number of problems and would like to deliver some new features to our users too.  Exciting!

Daily Value was resubmitted to Apple for approval with some bug fixes and new features.  I didn’t expect a quick turn-around this time, however.  Recent guidance from Apple mentioned that “90% of apps are approved within 14 days.”  It had only been about a week, but I received a rather vanilla email from the app review team that Daily Value “is requiring unexpected additional time for review.”  Not the kind of email to read before quitting for the day as my mind considered largely negative possibilities!

Unfortunately, the email is all of two lines and vaguely promises to “update you with further status as soon as we are able.”  So, I did a bunch of Google searches to see if others have encountered the same email and what the general results are – it seemed a largely mixed bag, but many of the posts were pretty old.  So, I thought I’d weigh in with my conjecture and update this post as I get more information.

In the earlier days (a year or so ago) of app development, this email seemed to indicate some sort of problem with your app – usually procedural.  Folks would use language, subject matter, or content that in Apple’s opinion was inappropriate.  In some cases, it appeared Apple may have held up an app if it may have encroached on another company’s trademarks or patents to not only protect themselves but the developer.

In fact earlier this month, Looking Glass submitted an app, iGive Blood, that not only received this same “unexpected time” email but was also rejected.  The problem was the app, and app store text mentioned that proceeds from the app’s sales would be donated to charity.  It wasn’t appropriate and the text had to be removed from the app and store text before being resubmitted.  No problem – a new binary was made and was resubmitted nine days ago.

Daily Value, however, was approved earlier in the month.  The changes in version 1.1 are feature enhancements, new functionality, and quality improvements – so, I’m curious about why it’s possibly being held up.

While I don’t have any concrete evidence other than anecdotal experience, I’d like to believe that the app review team is being flooded with apps.  When Daily Value first was being designed, there were fewer than 24,000 apps available – but four months later the number has reached nearly 65,000.  That’s not even counting updates and fixes to existing applications.  I thought I read on Apple’s web site that they get roughly 4,000 submissions (updates and new apps) per day.  Approval time has gone from 7-10 days to nearly 14.

Anyway, optimistically, I’m hoping that the form email is generated by an automated system after an app submission has been in the queue for a week – and with the deluge of apps to review and pour over – more and more developers will be receiving this email that is just serving notice “you app is in the queue – be patient” – and it’s not always as ill an omen as it used to be back when app approval was much quicker.  Fingers crossed, anyway!

Hello world!

Welcome to the Looking Glass development blog.

It’s my hope to share insight, challenges, and solutions to some of the problems I’ve faced while developing applications for Apple’s iPhone.

In many ways, the post title is quite appropriate.  There’s a lot of work involved in putting together a blog, web site, and most importantly – apps!  The first thing you do/try when learning a new language or environment is put together a simple test program.  While I’ve got a reasonable amount of experience as a software developer, web publishing and development is another thing entirely.

Time grants wisdom with practice and dedication – so, I’m optimistic with further posts and entries – the content of this blog will equally evolve and improve too!  Maybe when I get a break from app coding, I’ll poke about the blog framework and make some customizations… expect some changes!