Moves Import From SQLite Database (Instead of Export ZIP)

Hi, I used Moves app back in the day. Never got around to grab the export of my data. I do, however, have a copy of the SQLite database from an old iPhone backup.

I know that importing from the SQLite database is not possible. So I wrote a Python script that grabs data from the database and converts it into daily storyline JSON files. The result looks promising (excerpt below).

When I copy the JSON file into the Arc App iCloud folder I can see the import badge light up for a second but the data does not appear in the app timeline. (Tried to check the iPhone logs, too, but cannot make sense of the Arc logs.)

My question is: is there any place where I can read up on the file format, the directory structure and whatever else is expected from a moves_export.zip in order to replicate this as close as possible in hopes to get the old data imported?

What I tried:

  • added the storyline_YYYYMMDD.json file to the iCloud/Arc App/Import/ folder
  • added the storyline_YYYYMMDD.json file to the iCloud/Arc App/moves_export/json/daily/storyline/ folder
  • added a moves_export.zip file (with the json/daily/storyline/storyline_YYYYMMDD.json directory/file structure) to iCloud/Arc App/

The zip file is unpacked (by Arc, I assume). But that’s my only success up until here. As I don’t have a (somewhat) complete picture what’s expected by Arc for this to properly work, I am afraid I am stuck. Any kind of pointer, hint at a “official” documentation or just a (working and redacted example file) would be greatly appreciated.

Thanks in advance!

Generated JSON data (excerpt)

(lat, lon, date/time fields redacted)

{
  "date": "20131008",
  "summary": [
    {
      "activity": "walking",
      "group": "walking",
      "duration": 1561,
      "steps": 2587,
      "distance": 1783
    },
    {
      "activity": "transport",
      "group": "transport",
      "duration": 1539,
      "steps": 0,
      "distance": 7023
    }
  ],
  "segments": [
    {
      "type": "move",
      "activities": [
        {
          "totalDistanceLoc": 771.0,
          "trackPoints": [
            {
              "time": "20131008T102000+0200",
              "lat": 1.0,
              "hacc": 1,
              "lon": 1.0
            },
            {
              "time": "20131008T102000+0200",
              "lat": 1.0,
              "hacc": 16.0,
              "lon": 1.0
            },
            {
              "time": "20131008T102000+0200",
              "lat": 1.0,
              "hacc": 4.0,
              "lon": 1.0
            }
          ],
          "startTime": "20131008T102000+0200",
          "totalDistanceStep": 719,
          "steps": 959,
          "bridged": false,
          "filteredSteps": 0,
          "totalTime": 545,
          "activity": "walking",
          "totalDistance": 771,
          "group": "walking"
        }
      ],
      "startTime": "20131008T102000+0200",
      "steps": 0,
      "bridged": false,
      "endTime": "20131008T102000+0200"
    },
    {
      "bridged": false,
      "place": {
        "id": 917312,
        "type": "unknown",
        "location": {
          "lat": 1.0,
          "lon": 1.0
        }
      },
      "type": "place",
      "startTime": "20131008T102000+0200",
      "activities": [
        {
          "totalDistanceLoc": 1,
          "trackPoints": [
            {
              "time": "20131008T102000+0200",
              "lat": 1.0,
              "hacc": 100.0,
              "lon": 1.0
            },
            {
              "time": "20131008T102000+0200",
              "lat": 1.0,
              "hacc": 100.0,
              "lon": 1.0
            }
          ],
          "startTime": "20131008T102000+0200",
          "totalDistanceStep": 15,
          "steps": 20,
          "bridged": false,
          "filteredSteps": 0,
          "totalTime": 30,
          "activity": "walking",
          "totalDistance": 10,
          "group": "walking"
        }
      ],
      "location": {
        "lat": 1.0,
        "lon": 1.0
      },
      "endTime": "20131008T102000+0200",
      "steps": 0
    }
  ]
}

Hi @dan!

If the Importing badge shows up (albeit briefly) then I’m guessing you’re already putting the files in the right place, right filenames, etc.

So I suspect what’s happening is Arc’s Moves importer is broken. That code is over 6 years old now, and hasn’t been tested or updated in 4+ years. So yeah, my first guess at the most likely problem is that the code just isn’t working properly anymore, due to other things around it having subtly or significantly changed over the years.

Aside: I was actually going to rip that Moves importing code out completely recently. But figured it at least wasn’t causing any problems elsewhere in the system, and might still be working correctly, so no harm in leaving it. Though you might have proven now that it’s not still working correctly!

We could start diving in to the debugging process, to figure out what’s going wrong in the importer. There’s a chance it might just be one or two minor disconnects between old code and new, that can be patched up without too much fuss, if we can find them.

Though the other option is to convert your SQLite data to GPX instead of Moves storyline format. Arc’s GPX importer is very new, and much more advanced than the dodgy old Moves importer (that I wrote in couple of weeks, in a panic, back when Facebook started shutting down Moves with short notice).

Though GPX is inherently going to be more fiddly to generate, given that it’s both XML and a somewhat different structure from what’s going to be in the Moves SQLite db.

Let me know which way you want to approach it. If you want to go with the storyline format, feel free to send me some example files to matt@bigpaua.com, and I’ll see if I can get them importing on a test device here, and get that debugging process going.

The advantage of going the GPX route would be that you’d have fine grained control over what gets imported, with the ability to correct timezone issues and activity type classification issues at import time in the GPX importer’s advanced interface. The disadvantage would be the likely tedious process of twisting the Moves data into the right shapes to fit into GPX.

Both options sound equally viable to me, so I’m happy to assist with either!