My Journey Into Analyzing Apple Health Data

I’ve owned an Apple Watch for years – since the Series 2. I’ve also been running consistently for years – three of them, to be precise. I also like data. I’ve been collecting data on my workouts via my phone and watch for years, but getting data off of the iPhone’s small screen has always been problematic.

“But Peter!” you say “Apple lets you export data from the Health app!” Yes, it does. Have you ever looked at it? It looks something like this. Correction – it looks EXACTLY like this.

Well-structured XML

It’s XML data, and that doesn’t easily lend itself to a graph. Also, my data is over 1.3GB at present. That’s a lot of data for one guy. So I looked around for how to analyze my iOS Health data. The first site I found that looked promising was Analyze the Crap Out of Your Apple Health/HealthKit Data (sep.com) and GitHub – jonfuller/health-parse: Parses an Apple Health data export… for reasons. The developer offers an email address – email your health data to applehealthdata@gmail.com and it will send back parsed stuff. Sure. I’ll email you my 1.3GB of Apple Health Data. Let me know when you get it. Okay, no, I did not try that because it’s never going to work. So I downloaded his code from Github, but I couldn’t get it to compile. Seems like I’m not the only one, as others reported the same issue.

Next I tried the Heartwatch app for iOS. So close! It generates some nice reports but only goes back one year. I want to track data over multiple years. I emailed the developer, and he said he’d consider it.

Then I tried the YouTube video (1) How to download, graph and assess physical activity and exercise data from Apple Watch – YouTube. OMG hilarious. Fail. 

Something in Python perhaps?  Analyze Your iOS Health Data With Python | by Guido Casiraghi | Better Programming | Medium Prerequisites: You know the basics of Pandas. I don’t even know what pandas is, other than a big bear-like thing that lives in China. 

I tried to import the XML files into Excel. Hahahaha. I’m running the 32-bit version. It cannot open a 1.3GB XML file. 

I poked around and found this article by Taras Kaduk: Analyze and visualize your iPhone’s Health app data in R. I was told R is easy to learn and use, so I figured I’d give it a try.

I installed R for Windows. The UI seems a bit dated and barebones. The Comprehensive R Archive Network (case.edu) How do I install libraries, anyway?  HodentekHelp: How do you install the XML library for R programming? Okay, manual process, must select stuff from a list by point and click. Yuck. 

How do I change directories in R?  how to set path in R on Windows – Google Search

Hm. This looks kinda neat and more polished.   Download the RStudio IDE – RStudioHave to install those libraries, but at least I can type their names in a comma-separated list. Much quicker. 

How do I change directories in R again?  getwd, setwd | R Function of the Day

What’s the path to my files in my OneDrive folder without spaces in it?  Use PowerShell to display Short File and Folder Names | Scripting Blog (microsoft.com)

How do you comment in R? Comments in R – GeeksforGeeks

How do you print more lines than it’s showing me?  how to increase the limit for max.print in R – Stack Overflow

What does that %>% do?  Simplify Your Code with %>% · UC Business Analytics R Programming Guide (uc-r.github.io)

Oof. Guess I should take a lesson. Learn R | Codecademy

Yup, that did it! The following R code imports my Health XML data and spits out a CSV. And yeah, it took a lot of floundering to get these few lines of code:

library(XML)
library(dplyr)
xml <- xmlParse('export.xml')
df_workout <-  XML:::xmlAttrsToDataFrame(xml["//Workout"])
write_csv(df_workout,'health_export.csv')

Now I have a CSV file! Great! I’ll make a chart in Excel. OMFG Excel charting is beyond convoluted. Why is it so F***ING COMPLICATED?!?! 

Google Sheets to the rescue. Finally. I have what I have sought for months.

I realized that with the right libraries, I likely could have accomplished the same thing with Perl or Python, but learning R has been fun and I may have applications for this professionally as well as personally. Also, I should be able to generate the graphs directly from R, but haven’t learned that yet. Finally, I will likely need to dive deeper into the data to incorporate steps per minute and heartrate into the above chart. I’m really interested in overlaying my steps per minute and average heartrate to see how this affects energy used and pace. So while I’ve taken the first step (no pun intended), I’m not done yet!

180 Steps Per Minute

On my last two 10k runs, I experimented with upping my cadence (steps per minute). My normal cadence has been in the 155-165 steps per minute, and my normal pace has usually been in the high 9 to low 10 minutes per mile. This is mainly because I haven’t really cared to address my speed, so I would just ramble on at whatever pace I felt like running at, unless I was in a hurry or with someone else.

While I had heard that my pace “should be 180 steps per minute,” it was never clear to me as to why this was important. Additionally, given my training in yoga and anatomy, in which a recurring theme was “ever body is different,” how could it be that these two guys should be expected to have the same cadence?

Game of Thrones: Tyrion Lannister (Season 7) | 1:6 Scale Peter Dinklage |  Three Zero 903959
Tyrion
Gregor Clegane - Wikipedia
The Mountain

No, really. How can that make any sense? It should not be a surprise that there is some room for variation here, as with all things anatomy, and 180 is likely just an average. That said, I figured “what the heck? Why not give it a try.” After a little digging I found a website that suggested working your way up to 180 by starting with a playlist that runs around 10 bpm faster than your current average cadence. So I found a playlist on Apple Music that ran at 165 bpm, and I hit the road.

I immediately felt like I was moving faster – and I was. At first it was definitely more work, but after a while, I got into a rhythm. On a funny note, I found out after I finished my run that I had averaged 182 steps per minute! I knew that I was not keeping time with the beat of the music, but I didn’t realize it was because I was going too fast. Given that, I picked another playlist, this time running at 180 bpm. On my second attempt at keeping this cadence, I actually was a bit slower, coming down to the mid 170s on average. That said, it was still one of my fastest – if not the fastest – 10k I’ve done, coming in with an average of 8’59”.

In short, I’m sold. I’m going to try to keep up with the 180 steps per minute cadence. Supposedly this is universal, and applies to all forms of terrain. Time will tell as I experiment. Watch this space for an update!

Running Update – Half Marathon

Wow. Over a year between blog posts. I’m on a roll! I forgot to brag that, in March, I ran 13.1 miles. That’s the equivalent of running from Athens to Kifisia, Greece. For those of you who are cartographically-challenged, Kifisia is halfway between Athens and Maration, which is 26.2 miles away. Is it starting to make sense now?

In February, I started adding one mile to my runs. Every week, I would run two or three times, adding a mile each week. When I started this, I had been doing my semi-regular 5 miles per run. The first week, starting on February 2, I ran 7 miles three times. The next week I ran 8 miles twice. The next week was 9, twice, followed by a 10 miler (my first since 2019), then 11 miles, then 12, and then finally, on March 4 I hit 13.21 miles. My average pace was 10’23”, which is on my higher end of pavement running these days, so I am still pretty happy with the numbers.

Since then, I’ve done a couple of 8 mile runs, but pretty have much standardized on 6.25 (10k) every other day. The heat definitely affects me, and when it’s in the high 80s or more, I generally skip it or go trail riding. Trail riding has also cut into my running routine somewhat, but it’s also fun, good exercise, and gets me outside, so no foul!

How I Went From Couch to 5k, then 5 Miles 10kMiles

In the fall of 2017 I had a roommate. He’s a runner. Like, a serious runner. You know, the kind who goes out for a 5 mile run on one of his rest days? The kind who’s training for a 50 miler? The kind who has to eat 8,000 calories a day to maintain his weight? That kind.

One day (October 27, to be precise) he asked if I’d like to go out for a “recovery run” with him.

“How long?” I asked.

Just 2 miles,” he replied.

“Sure,” I answered, expecting it would suck, but being in relatively decent shape to begin with, why not?

It sucked, that’s why not! Ugh. It was grueling. I managed to maintain an average pace of 8’55”, which I thought was pretty decent.

A couple of days later, I decided to try again, only I would start smaller – say, half a mile. I did .43 miles in 9’10”. Not great! But I was alone and didn’t have my coach kicking me every step of the way.

The next day I did it again. .42 miles at 8’57”.

The next day I did it again. 1.01 miles at 0’03”.

From there I dabbled, going back and forth with a half mile some days, 2 miles the next. I didn’t have a plan, other than “run a mile,” which I did every couple of days. I continued that until June, when I decided I’d just start adding a little each day. Each time I ran, I added .1 or so miles until I hit 3.74 in July (and that day was a hot, sweaty, doozy).

I’d looked at formal plans in the past, and never really had much luck with making them stick. I even tried to go onto the “couch to 5k plan” to see how that would help me, until I realized that 5k is just 3.1 miles, and I was already there, so…

On April 6, 2019, I decided to kick it up a notch and shoot for 5 miles. Again, I decided to just add a little every time I ran. Since I was averaging a run every other day, I’d add .2 miles each time. I went from 3.1, to 3.2, to 3.3, 3.5, 3.7, 3.8, 4.0, 4.2, 4.4, 4.6, and finally 5.0 exactly 30 days later.

Since then, I’ve set 5 miles as my standard, shooting for a run every 2 days. I’ve done four of them so far and today will be my fifth.

Nike may not like it, but in summary, my plan is “just do it.” Add a little bit each day, making slow and steady progress. If you backslide, don’t worry about it! Just get out and do it again. Before you know it, a 5 k run will be just a warmup, and it taking supreme effort will be just a memory.

Update: On Saturday, May 25, 2019 I took the next step and upped my run to 10k. I’ve done it twice and plan on making this a regular workout a 2-3 times a week, work and weather permitting.

Update: On Saturday, June 1, 2019 I took the next step and upped my run to 10 miles.