Skip to content

Instantly share code, notes, and snippets.

@mattharrison

mattharrison/accelerate-w-polars.ipynb

Created February 7, 2025 22:19
Show Gist options
  • Save mattharrison/0e2b7fb295e2f16558bf20214fed5129 to your computer and use it in GitHub Desktop.
Save mattharrison/0e2b7fb295e2f16558bf20214fed5129 to your computer and use it in GitHub Desktop.
Download ZIP
accelerate-w-polars.ipynb
Display the source blob
Display the rendered blob
Raw
accelerate-w-polars.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/mattharrison/0e2b7fb295e2f16558bf20214fed5129/accelerate-w-polars.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "aawghS8mb7DQ"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"source": [
"## Install Accelerator\n",
"\n",
"I'm using an L4"
],
"metadata": {
"id": "n7VzCZm3b71M"
}
},
{
"cell_type": "code",
"source": [
"!nvidia-smi | head"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "dKoMnjG2b-gU",
"outputId": "e0b155b8-f0be-461b-fb3e-d08a0b342eb1"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Sat Feb 1 21:35:14 2025 \n",
"+-----------------------------------------------------------------------------------------+\n",
"| NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 |\n",
"|-----------------------------------------+------------------------+----------------------+\n",
"| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |\n",
"| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |\n",
"| | | MIG M. |\n",
"|=========================================+========================+======================|\n",
"| 0 NVIDIA L4 Off | 00000000:00:03.0 Off | 0 |\n",
"| N/A 44C P8 12W / 72W | 0MiB / 23034MiB | 0% Default |\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!uv pip install --system -U --extra-index-url=https://pypi.nvidia.com polars[gpu] cudf-polars-cu12"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "ZAA-HavTcGna",
"outputId": "e53f31ae-71ee-4bdc-8bf4-2b244566f90c"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"\u001b[2mUsing Python 3.11.11 environment at /usr\u001b[0m\n",
"\u001b[2K\u001b[2mResolved \u001b[1m15 packages\u001b[0m \u001b[2min 9.33s\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/77.59 KiB\n",
"\u001b[2K\u001b[2A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2K\u001b[2A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/15.58 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/33.88 MiB\n",
"\u001b[2K\u001b[4A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/33.88 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m\u001b[2m------------------------------\u001b[0m\u001b[0m 0 B/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 16.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 32.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 32.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 48.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 32.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 64.00 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 16.00 KiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mcudf-polars-cu12\u001b[0m \u001b[32m-------------------------\u001b[2m-----\u001b[0m\u001b[0m 64.00 KiB/77.59 KiB\n",
"\u001b[2mnumba \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 430.18 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 32.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 96.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 1.05 MiB/38.22 MiB\n",
"\u001b[2K\u001b[6A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 525.85 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 176.00 KiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 143.89 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 1.89 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 653.16 KiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 1.08 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 175.21 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--\u001b[2m----------------------------\u001b[0m\u001b[0m 2.17 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m---------\u001b[2m---------------------\u001b[0m\u001b[0m 1.08 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.84 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-\u001b[2m-----------------------------\u001b[0m\u001b[0m 432.00 KiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 2.86 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (0/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 1.16 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.91 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 3.00 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m---\u001b[2m---------------------------\u001b[0m\u001b[0m 3.45 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠹\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 1.20 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.91 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 4.57 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 4.33 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠹\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 1.25 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.91 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 5.70 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 5.92 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠹\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 1.28 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.92 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 6.92 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 7.43 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠹\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 1.31 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.94 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m--------\u001b[2m----------------------\u001b[0m\u001b[0m 8.12 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-------\u001b[2m-----------------------\u001b[0m\u001b[0m 8.68 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠸\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------------\u001b[2m------------------\u001b[0m\u001b[0m 1.34 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.95 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m---------\u001b[2m---------------------\u001b[0m\u001b[0m 9.61 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--------\u001b[2m----------------------\u001b[0m\u001b[0m 10.10 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠸\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------------\u001b[2m------------------\u001b[0m\u001b[0m 1.36 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 1.97 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 10.94 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m---------\u001b[2m---------------------\u001b[0m\u001b[0m 11.27 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠸\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------------\u001b[2m------------------\u001b[0m\u001b[0m 1.45 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----\u001b[2m--------------------------\u001b[0m\u001b[0m 2.03 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 11.92 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 12.42 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠸\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.47 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.11 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m------------\u001b[2m------------------\u001b[0m\u001b[0m 13.23 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------\u001b[2m-------------------\u001b[0m\u001b[0m 13.84 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠼\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.48 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.12 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 14.55 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m------------\u001b[2m------------------\u001b[0m\u001b[0m 15.12 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠼\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.48 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.13 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m---------------\u001b[2m---------------\u001b[0m\u001b[0m 15.86 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 16.41 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠼\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.48 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.14 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------\u001b[2m--------------\u001b[0m\u001b[0m 17.08 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 17.81 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠴\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.50 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.16 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-----------------\u001b[2m-------------\u001b[0m\u001b[0m 18.50 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m----------------\u001b[2m--------------\u001b[0m\u001b[0m 19.19 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠴\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.52 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m------------------\u001b[2m------------\u001b[0m\u001b[0m 19.73 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------\u001b[2m-------------\u001b[0m\u001b[0m 20.55 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠴\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.52 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-------------------\u001b[2m-----------\u001b[0m\u001b[0m 21.03 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m------------------\u001b[2m------------\u001b[0m\u001b[0m 21.86 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠴\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.53 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 22.47 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-------------------\u001b[2m-----------\u001b[0m\u001b[0m 23.08 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠴\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.55 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 23.83 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 24.36 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠦\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.56 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 24.00 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m---------------------\u001b[2m---------\u001b[0m\u001b[0m 26.72 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠦\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-------------\u001b[2m-----------------\u001b[0m\u001b[0m 1.58 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.17 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 24.00 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------\u001b[2m-------\u001b[0m\u001b[0m 28.81 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠦\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 1.59 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.19 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 24.00 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 30.30 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠧\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 1.61 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.19 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 24.00 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m--------------------------\u001b[2m----\u001b[0m\u001b[0m 32.30 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠧\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 1.62 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.19 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------\u001b[2m--------\u001b[0m\u001b[0m 24.02 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m---------------------------\u001b[2m---\u001b[0m\u001b[0m 34.37 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠧\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 1.62 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.19 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-----------------------\u001b[2m-------\u001b[0m\u001b[0m 25.52 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 35.59 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠧\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 1.64 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m-----\u001b[2m-------------------------\u001b[0m\u001b[0m 2.20 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 27.05 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------------\u001b[2m-\u001b[0m\u001b[0m 36.25 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠧\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m--------------------\u001b[2m----------\u001b[0m\u001b[0m 2.43 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 2.61 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m-------------------------\u001b[2m-----\u001b[0m\u001b[0m 27.80 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------------\u001b[2m-\u001b[0m\u001b[0m 36.44 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m-----------------------\u001b[2m-------\u001b[0m\u001b[0m 2.75 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 2.75 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m--------------------------\u001b[2m----\u001b[0m\u001b[0m 29.31 MiB/33.88 MiB\n",
"\u001b[2mpyarrow \u001b[0m \u001b[32m-----------------------------\u001b[2m-\u001b[0m\u001b[0m 36.48 MiB/38.22 MiB\n",
"\u001b[2K\u001b[5A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 2.82 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 2.78 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[4A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m------------------------\u001b[2m------\u001b[0m\u001b[0m 2.84 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m------\u001b[2m------------------------\u001b[0m\u001b[0m 2.78 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[4A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumba \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 3.33 MiB/3.65 MiB\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m----------\u001b[2m--------------------\u001b[0m\u001b[0m 4.92 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[4A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 7.05 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠇\u001b[0m \u001b[2mPreparing packages...\u001b[0m (1/6)\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m--------------\u001b[2m----------------\u001b[0m\u001b[0m 7.05 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠋\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/6)\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m------------------\u001b[2m------------\u001b[0m\u001b[0m 9.23 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠋\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/6)\n",
"\u001b[2mnumpy \u001b[0m \u001b[32m--------------------------\u001b[2m----\u001b[0m\u001b[0m 13.08 MiB/15.58 MiB\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[3A\u001b[37m⠋\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/6)\n",
"\u001b[2mpolars \u001b[0m \u001b[32m----------------------------\u001b[2m--\u001b[0m\u001b[0m 30.97 MiB/33.88 MiB\n",
"\u001b[2K\u001b[2A\u001b[37m⠋\u001b[0m \u001b[2mPreparing packages...\u001b[0m (3/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[1A\u001b[37m⠙\u001b[0m \u001b[2mPreparing packages...\u001b[0m (5/6)\n",
"\u001b[2K\u001b[2mPrepared \u001b[1m6 packages\u001b[0m \u001b[2min 2.03s\u001b[0m\u001b[0m\n",
"\u001b[2mUninstalled \u001b[1m5 packages\u001b[0m \u001b[2min 140ms\u001b[0m\u001b[0m\n",
"\u001b[2K\u001b[2mInstalled \u001b[1m6 packages\u001b[0m \u001b[2min 23ms\u001b[0m\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mcudf-polars-cu12\u001b[0m\u001b[2m==24.12.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mllvmlite\u001b[0m\u001b[2m==0.43.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mllvmlite\u001b[0m\u001b[2m==0.44.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mnumba\u001b[0m\u001b[2m==0.60.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mnumba\u001b[0m\u001b[2m==0.61.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mnumpy\u001b[0m\u001b[2m==1.26.4\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mnumpy\u001b[0m\u001b[2m==2.1.3\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mpolars\u001b[0m\u001b[2m==1.9.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpolars\u001b[0m\u001b[2m==1.14.0\u001b[0m\n",
" \u001b[31m-\u001b[39m \u001b[1mpyarrow\u001b[0m\u001b[2m==17.0.0\u001b[0m\n",
" \u001b[32m+\u001b[39m \u001b[1mpyarrow\u001b[0m\u001b[2m==18.1.0\u001b[0m\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"!ls"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "Fis0cHU4cOg3",
"outputId": "2f44d32f-ace5-4ea8-fdf3-f8dd0ba607bc"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"sample_data\n"
]
}
]
},
{
"cell_type": "code",
"source": [
"from google.colab import drive\n",
"drive.mount('/content/drive')"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "968gae16cXvY",
"outputId": "88544d9d-797c-4f9d-96e2-65d00161d59f"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"Mounted at /content/drive\n"
]
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "d9LYQMCab0QS",
"outputId": "fdf893ab-0e85-4de2-b068-294533ce6580"
},
"outputs": [
{
"data": {
"text/plain": [
"'1.21.0'"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import polars as pl\n",
"pl.__version__"
]
},
{
"cell_type": "code",
"source": [
"!ls /content/drive/MyDrive/Colab\\ Notebooks/clients/nvidia/GTC2024"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "W3mQ91Mccs4V",
"outputId": "1bc15ac2-612b-4524-8346-f4f076159302"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"accelerate-w-polars.ipynb data\n"
]
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "XYVbxPvab0QT"
},
"source": [
"## Getting Data\n",
"\n",
"If you threaten to delete your Strava account, they will let you download all your data [^q]\n",
"\n",
"[^q]: https://support.strava.com/hc/en-us/articles/216918437-Exporting-your-Data-and-Bulk-Export"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "SqCzOxCjb0QU",
"outputId": "6e7df78c-a79c-44c9-b706-bac5a216499c"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"total 118121\n",
"-rw------- 1 root root 120955549 Feb 1 01:40 activities.parquet\n"
]
}
],
"source": [
"!ls -l /content/drive/MyDrive/Colab\\ Notebooks/clients/nvidia/GTC2024/data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VJ9WELGIb0QU",
"outputId": "7d780e06-f0ab-437a-f20b-82ea296229f7"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Activity ID,Activity Date,Activity Name,Activity Type,Activity Description,Elapsed Time,Distance,Max Heart Rate,Relative Effort,Commute,Activity Private Note,Activity Gear,Filename,Athlete Weight,Bike Weight,Elapsed Time,Moving Time,Distance,Max Speed,Average Speed,Elevation Gain,Elevation Loss,Elevation Low,Elevation High,Max Grade,Average Grade,Average Positive Grade,Average Negative Grade,Max Cadence,Average Cadence,Max Heart Rate,Average Heart Rate,Max Watts,Average Watts,Calories,Max Temperature,Average Temperature,Relative Effort,Total Work,Number of Runs,Uphill Time,Downhill Time,Other Time,Perceived Exertion,Type,Start Time,Weighted Average Power,Power Count,Prefer Perceived Exertion,Perceived Relative Effort,Commute,Total Weight Lifted,From Upload,Grade Adjusted Distance,Weather Observation Time,Weather Condition,Weather Temperature,Apparent Temperature,Dewpoint,Humidity,Weather Pressure,Wind Speed,Wind Gust,Wind Bearing,Precipitation Intensity,Sunrise Time,Sunset Time,Moon Phase,Bike,Gear,Precipitation Probability,Precipitation Type,Cloud Cover,Weather Visibility,UV Index,Weather Ozone,Jump Count,Total Grit,Average Flow,Flagged,Average Elapsed Speed,Dirt Distance,Newly Explored Distance,Newly Explored Dirt Distance,Activity Count,Total Steps,Carbon Saved,Pool Length,Training Load,Intensity,Average Grade Adjusted Pace,Timer Time,Total Cycles,Media\n",
"484762708,\"Jan 1, 2016, 9:59:32 PM\",Mueller Park Snowshoe,Snowshoe,w/ family,9797,3.28,,,,,,activities/536281923.gpx.gz,81.64659881591797,,9797.0,3082.0,3282.39990234375,3.0,,105.38999938964844,,1586.699951171875,1694.199951171875,35.099998474121094,0.006093100178986788,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762713,\"Jan 11, 2016, 8:35:00 PM\",Mueller Park Ski,Backcountry Ski,With Car at end,5438,8.34,,,,,,activities/536281927.gpx.gz,81.64659881591797,,5438.0,3492.0,8344.599609375,21.700000762939453,,274.0150146484375,,1381.800048828125,1871.5,47.79999923706055,-2.550149917602539,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762714,\"Jan 16, 2016, 2:53:15 PM\",Mueller Park Ski,Backcountry Ski,\"\",6963,5.24,,,,,,activities/536281925.gpx.gz,81.64659881591797,,6963.0,3915.0,5241.2001953125,7.699999809265137,,284.45599365234375,,1592.699951171875,1882.800048828125,48.5,0.032435301691293716,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762723,\"Jan 2, 2016, 4:44:20 PM\",Mueller Park Run,Run,With Tyler,5248,10.89,,,false,,Luna Sandals Oso,activities/536281928.gpx.gz,81.64659881591797,,5248.0,4619.0,10894.2001953125,7.5,,421.48199462890625,,1585.0999755859375,1906.0,47.79999923706055,0.07343360036611557,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,1225675.0,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762726,\"Dec 23, 2013, 2:03:46 PM\",Tom's hill,Backcountry Ski,Tom's hill,12289,8.78,,,,,,activities/536281939.gpx.gz,81.64659881591797,,12289.0,7478.0,8782.900390625,10.5,,650.3150024414062,,2237.300048828125,2766.89990234375,48.599998474121094,0.006831460166722536,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762736,\"Apr 18, 2014, 7:09:10 PM\",Snowbird,Alpine Ski,\"\",10885,33.70,,,,,,activities/536281936.gpx.gz,81.64659881591797,,10885.0,7482.0,33706.5,18.799999237060547,,4017.570068359375,,2401.199951171875,3347.0,50.0,-0.1741500049829483,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762737,\"Jan 25, 2016, 4:25:01 PM\",Park City,Alpine Ski,\"\",9729,37.83,,,,,,activities/536281931.gpx.gz,81.64659881591797,,9729.0,8799.0,37836.30078125,19.0,,3574.889892578125,,2101.10009765625,2948.5,49.79999923706055,0.0007928889826871455,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762739,\"Jan 15, 2016, 4:39:51 PM\",Park City,Alpine Ski,Solo,9139,37.05,,,,,,activities/536281932.gpx.gz,81.64659881591797,,9139.0,7788.0,37057.5,15.699999809265137,,3085.389892578125,,2101.300048828125,2956.0,49.70000076293945,0.0008095530210994184,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n",
"484762742,\"Jan 5, 2016, 4:36:26 PM\",Canyons,Alpine Ski,With Wilsons,26401,57.76,,,,,,activities/536281921.gpx.gz,81.64659881591797,,26401.0,12845.0,57767.3984375,16.799999237060547,,5239.39990234375,,2107.800048828125,2833.0,50.0,-0.058856699615716934,,,,,,,,,,,,,,,,,,,,,,,,,0.0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,\"\"\n"
]
}
],
"source": [
"!head data/activities.csv"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-3x6vWHNb0QU",
"outputId": "db0aa9d4-3b94-4d35-cb0f-edd8abc92637"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (990, 94)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>Activity ID</th><th>Activity Date</th><th>Activity Name</th><th>Activity Type</th><th>Activity Description</th><th>Elapsed Time</th><th>Distance</th><th>Max Heart Rate</th><th>Relative Effort</th><th>Commute</th><th>Activity Private Note</th><th>Activity Gear</th><th>Filename</th><th>Athlete Weight</th><th>Bike Weight</th><th>Elapsed Time_duplicated_0</th><th>Moving Time</th><th>Distance_duplicated_0</th><th>Max Speed</th><th>Average Speed</th><th>Elevation Gain</th><th>Elevation Loss</th><th>Elevation Low</th><th>Elevation High</th><th>Max Grade</th><th>Average Grade</th><th>Average Positive Grade</th><th>Average Negative Grade</th><th>Max Cadence</th><th>Average Cadence</th><th>Max Heart Rate_duplicated_0</th><th>Average Heart Rate</th><th>Max Watts</th><th>Average Watts</th><th>Calories</th><th>Max Temperature</th><th>Average Temperature</th><th>&hellip;</th><th>Apparent Temperature</th><th>Dewpoint</th><th>Humidity</th><th>Weather Pressure</th><th>Wind Speed</th><th>Wind Gust</th><th>Wind Bearing</th><th>Precipitation Intensity</th><th>Sunrise Time</th><th>Sunset Time</th><th>Moon Phase</th><th>Bike</th><th>Gear</th><th>Precipitation Probability</th><th>Precipitation Type</th><th>Cloud Cover</th><th>Weather Visibility</th><th>UV Index</th><th>Weather Ozone</th><th>Jump Count</th><th>Total Grit</th><th>Average Flow</th><th>Flagged</th><th>Average Elapsed Speed</th><th>Dirt Distance</th><th>Newly Explored Distance</th><th>Newly Explored Dirt Distance</th><th>Activity Count</th><th>Total Steps</th><th>Carbon Saved</th><th>Pool Length</th><th>Training Load</th><th>Intensity</th><th>Average Grade Adjusted Pace</th><th>Timer Time</th><th>Total Cycles</th><th>Media</th></tr><tr><td>i64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>f64</td><td>str</td><td>str</td><td>bool</td><td>str</td><td>str</td><td>str</td><td>f64</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>str</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>&hellip;</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>f64</td><td>str</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>484762708</td><td>&quot;Jan 1, 2016, 9:59:32 PM&quot;</td><td>&quot;Mueller Park Snowshoe&quot;</td><td>&quot;Snowshoe&quot;</td><td>&quot;w/ family&quot;</td><td>9797</td><td>3.28</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;activities/536281923.gpx.gz&quot;</td><td>81.646599</td><td>null</td><td>9797.0</td><td>3082.0</td><td>3282.399902</td><td>3.0</td><td>null</td><td>105.389999</td><td>null</td><td>1586.699951</td><td>1694.199951</td><td>35.099998</td><td>0.006093</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>484762713</td><td>&quot;Jan 11, 2016, 8:35:00 PM&quot;</td><td>&quot;Mueller Park Ski&quot;</td><td>&quot;Backcountry Ski&quot;</td><td>&quot;With Car at end&quot;</td><td>5438</td><td>8.34</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;activities/536281927.gpx.gz&quot;</td><td>81.646599</td><td>null</td><td>5438.0</td><td>3492.0</td><td>8344.599609</td><td>21.700001</td><td>null</td><td>274.015015</td><td>null</td><td>1381.800049</td><td>1871.5</td><td>47.799999</td><td>-2.55015</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>484762714</td><td>&quot;Jan 16, 2016, 2:53:15 PM&quot;</td><td>&quot;Mueller Park Ski&quot;</td><td>&quot;Backcountry Ski&quot;</td><td>&quot;&quot;</td><td>6963</td><td>5.24</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;activities/536281925.gpx.gz&quot;</td><td>81.646599</td><td>null</td><td>6963.0</td><td>3915.0</td><td>5241.200195</td><td>7.7</td><td>null</td><td>284.455994</td><td>null</td><td>1592.699951</td><td>1882.800049</td><td>48.5</td><td>0.032435</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>484762723</td><td>&quot;Jan 2, 2016, 4:44:20 PM&quot;</td><td>&quot;Mueller Park Run&quot;</td><td>&quot;Run&quot;</td><td>&quot;With Tyler&quot;</td><td>5248</td><td>10.89</td><td>null</td><td>null</td><td>false</td><td>null</td><td>&quot;Luna Sandals Oso&quot;</td><td>&quot;activities/536281928.gpx.gz&quot;</td><td>81.646599</td><td>null</td><td>5248.0</td><td>4619.0</td><td>10894.200195</td><td>7.5</td><td>null</td><td>421.481995</td><td>null</td><td>1585.099976</td><td>1906.0</td><td>47.799999</td><td>0.073434</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>1.225675e6</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>484762726</td><td>&quot;Dec 23, 2013, 2:03:46 PM&quot;</td><td>&quot;Tom&#x27;s hill&quot;</td><td>&quot;Backcountry Ski&quot;</td><td>&quot;Tom&#x27;s hill&quot;</td><td>12289</td><td>8.78</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;activities/536281939.gpx.gz&quot;</td><td>81.646599</td><td>null</td><td>12289.0</td><td>7478.0</td><td>8782.900391</td><td>10.5</td><td>null</td><td>650.315002</td><td>null</td><td>2237.300049</td><td>2766.899902</td><td>48.599998</td><td>0.006831</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>13375285162</td><td>&quot;Jan 16, 2025, 11:47:52 PM&quot;</td><td>&quot;Evening walk&quot;</td><td>&quot;Walk&quot;</td><td>null</td><td>4400</td><td>3.73</td><td>null</td><td>null</td><td>false</td><td>null</td><td>&quot;Luna Sandals Oso&quot;</td><td>&quot;activities/13375285162.gpx&quot;</td><td>null</td><td>null</td><td>4400.0</td><td>3123.0</td><td>3734.699951</td><td>2.793333</td><td>&quot;1.1958693265914917&quot;</td><td>1077.065674</td><td>1077.065674</td><td>-245.399994</td><td>460.200012</td><td>50.0</td><td>-0.002678</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>374.655945</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>1.225675e6</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0.0</td><td>&quot;0.848795473575592&quot;</td><td>2143.800049</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>13381425823</td><td>&quot;Jan 17, 2025, 7:17:10 PM&quot;</td><td>&quot;Morning walk&quot;</td><td>&quot;Walk&quot;</td><td>null</td><td>1567</td><td>2.08</td><td>null</td><td>null</td><td>false</td><td>null</td><td>&quot;Luna Sandals Oso&quot;</td><td>&quot;activities/13381425823.gpx&quot;</td><td>null</td><td>null</td><td>1567.0</td><td>1428.0</td><td>2087.300049</td><td>2.98</td><td>&quot;1.4616947174072266&quot;</td><td>26.953091</td><td>26.953091</td><td>4.9</td><td>22.9</td><td>22.38806</td><td>-0.095818</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>209.392807</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>1.225675e6</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0.0</td><td>&quot;1.332035779953003&quot;</td><td>1811.300049</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>13391655502</td><td>&quot;Jan 18, 2025, 7:32:25 PM&quot;</td><td>&quot;Basin 13&quot;</td><td>&quot;Alpine Ski&quot;</td><td>null</td><td>10485</td><td>38.82</td><td>null</td><td>null</td><td>false</td><td>null</td><td>null</td><td>&quot;activities/13391655502.gpx&quot;</td><td>null</td><td>null</td><td>10485.0</td><td>7920.0</td><td>38826.199219</td><td>19.719999</td><td>&quot;4.9022979736328125&quot;</td><td>4198.770508</td><td>4198.790039</td><td>1948.099976</td><td>2692.800049</td><td>50.0</td><td>0.008242</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0.0</td><td>&quot;3.7030234336853027&quot;</td><td>4177.5</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>13408704724</td><td>&quot;Jan 20, 2025, 3:46:17 PM&quot;</td><td>&quot;DV 14&quot;</td><td>&quot;Alpine Ski&quot;</td><td>null</td><td>19235</td><td>51.45</td><td>null</td><td>null</td><td>false</td><td>null</td><td>null</td><td>&quot;activities/13408704724.gpx&quot;</td><td>null</td><td>null</td><td>19235.0</td><td>11398.0</td><td>51452.300781</td><td>23.74</td><td>&quot;4.514151573181152&quot;</td><td>5577.28125</td><td>5577.28125</td><td>2185.199951</td><td>2915.600098</td><td>50.0</td><td>-0.005831</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0.0</td><td>&quot;2.674931049346924&quot;</td><td>7265.899902</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr><tr><td>13480080256</td><td>&quot;Jan 28, 2025, 5:22:45 PM&quot;</td><td>&quot;Brighton 15&quot;</td><td>&quot;Alpine Ski&quot;</td><td>null</td><td>31277</td><td>71.92</td><td>null</td><td>null</td><td>false</td><td>null</td><td>null</td><td>&quot;activities/13480080256.gpx&quot;</td><td>null</td><td>null</td><td>31277.0</td><td>14011.0</td><td>71927.203125</td><td>24.469999</td><td>&quot;5.1336236000061035&quot;</td><td>6887.742188</td><td>6887.742188</td><td>2668.600098</td><td>3195.5</td><td>50.0</td><td>0.012791</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&hellip;</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>0.0</td><td>&quot;2.2996835708618164&quot;</td><td>12016.700195</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>null</td><td>&quot;&quot;</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (990, 94)\n",
"┌──────────────┬──────────────┬──────────────┬─────────────┬───┬──────────┬───────┬────────┬───────┐\n",
"│ Activity ID ┆ Activity ┆ Activity ┆ Activity ┆ … ┆ Average ┆ Timer ┆ Total ┆ Media │\n",
"│ --- ┆ Date ┆ Name ┆ Type ┆ ┆ Grade ┆ Time ┆ Cycles ┆ --- │\n",
"│ i64 ┆ --- ┆ --- ┆ --- ┆ ┆ Adjusted ┆ --- ┆ --- ┆ str │\n",
"│ ┆ str ┆ str ┆ str ┆ ┆ Pace ┆ str ┆ str ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ --- ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ str ┆ ┆ ┆ │\n",
"╞══════════════╪══════════════╪══════════════╪═════════════╪═══╪══════════╪═══════╪════════╪═══════╡\n",
"│ 484762708 ┆ Jan 1, 2016, ┆ Mueller Park ┆ Snowshoe ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 9:59:32 PM ┆ Snowshoe ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 484762713 ┆ Jan 11, ┆ Mueller Park ┆ Backcountry ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2016, ┆ Ski ┆ Ski ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 8:35:00 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 484762714 ┆ Jan 16, ┆ Mueller Park ┆ Backcountry ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2016, ┆ Ski ┆ Ski ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 2:53:15 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 484762723 ┆ Jan 2, 2016, ┆ Mueller Park ┆ Run ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 4:44:20 PM ┆ Run ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 484762726 ┆ Dec 23, ┆ Tom's hill ┆ Backcountry ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2013, ┆ ┆ Ski ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 2:03:46 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 13375285162 ┆ Jan 16, ┆ Evening walk ┆ Walk ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2025, ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 11:47:52 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13381425823 ┆ Jan 17, ┆ Morning walk ┆ Walk ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2025, ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 7:17:10 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13391655502 ┆ Jan 18, ┆ Basin 13 ┆ Alpine Ski ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2025, ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 7:32:25 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13408704724 ┆ Jan 20, ┆ DV 14 ┆ Alpine Ski ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2025, ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 3:46:17 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13480080256 ┆ Jan 28, ┆ Brighton 15 ┆ Alpine Ski ┆ … ┆ null ┆ null ┆ null ┆ │\n",
"│ ┆ 2025, ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ 5:22:45 PM ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"└──────────────┴──────────────┴──────────────┴─────────────┴───┴──────────┴───────┴────────┴───────┘"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pl.read_csv('data/activities.csv')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8T89B1ePb0QU"
},
"outputs": [],
"source": [
"import gpxpy\n",
"\n",
"data = gpxpy.parse(open('data/activities/13480080256.gpx'))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "nYUA3mx1b0QU",
"outputId": "b8c18fcf-d682-4afc-8b0a-bf6c63f6268c"
},
"outputs": [
{
"data": {
"text/plain": [
"'Brighton 15'"
]
},
"execution_count": 62,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data.tracks[0].name"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LEav63jWb0QU",
"outputId": "fa535f14-d2c6-419d-d469-4b790e038aa1"
},
"outputs": [
{
"data": {
"text/plain": [
"['__class__',\n",
" '__delattr__',\n",
" '__dir__',\n",
" '__doc__',\n",
" '__eq__',\n",
" '__format__',\n",
" '__ge__',\n",
" '__getattribute__',\n",
" '__gt__',\n",
" '__hash__',\n",
" '__init__',\n",
" '__init_subclass__',\n",
" '__le__',\n",
" '__lt__',\n",
" '__module__',\n",
" '__ne__',\n",
" '__new__',\n",
" '__reduce__',\n",
" '__reduce_ex__',\n",
" '__repr__',\n",
" '__setattr__',\n",
" '__sizeof__',\n",
" '__slotnames__',\n",
" '__slots__',\n",
" '__str__',\n",
" '__subclasshook__',\n",
" 'add_elevation',\n",
" 'add_missing_data',\n",
" 'adjust_time',\n",
" 'clone',\n",
" 'comment',\n",
" 'description',\n",
" 'extensions',\n",
" 'get_bounds',\n",
" 'get_center',\n",
" 'get_duration',\n",
" 'get_elevation_extremes',\n",
" 'get_location_at',\n",
" 'get_moving_data',\n",
" 'get_nearest_location',\n",
" 'get_points_no',\n",
" 'get_time_bounds',\n",
" 'get_uphill_downhill',\n",
" 'gpx_10_fields',\n",
" 'gpx_11_fields',\n",
" 'has_elevations',\n",
" 'has_times',\n",
" 'join',\n",
" 'length_2d',\n",
" 'length_3d',\n",
" 'link',\n",
" 'link_text',\n",
" 'link_type',\n",
" 'move',\n",
" 'name',\n",
" 'number',\n",
" 'reduce_points',\n",
" 'remove_elevation',\n",
" 'remove_empty',\n",
" 'remove_time',\n",
" 'segments',\n",
" 'simplify',\n",
" 'smooth',\n",
" 'source',\n",
" 'split',\n",
" 'type',\n",
" 'walk']"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dir(data.tracks[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6YbJFYyhb0QU",
"outputId": "7f88c026-6682-4140-872a-1f6cad580ec5"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"shape: (30_547, 10)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬───────────┬───────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ speed_bet ┆ name ┆ type ┆ filename │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ ween ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ --- ┆ str ┆ str ┆ str │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ f64 ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 40.59895 ┆ -111.58352 ┆ … ┆ null ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 7.527056 ┆ 40.599017 ┆ -111.58351 ┆ … ┆ 7.527056 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 3.311347 ┆ 40.599037 ┆ -111.58348 ┆ … ┆ 3.311347 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 1.287244 ┆ 40.599035 ┆ -111.58346 ┆ … ┆ 1.287244 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 1.325377 ┆ 40.599024 ┆ -111.58346 ┆ … ┆ 1.325377 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.0 ┆ 40.598024 ┆ -111.58345 ┆ … ┆ 0.0 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 0.0 ┆ 40.598024 ┆ -111.58345 ┆ … ┆ 0.0 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 0.0 ┆ 40.598024 ┆ -111.58345 ┆ … ┆ 0.0 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 0.0 ┆ 40.598024 ┆ -111.58345 ┆ … ┆ 0.0 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 0.0 ┆ 40.598024 ┆ -111.58345 ┆ … ┆ 0.0 ┆ Brighton ┆ AlpineSki ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ 15 ┆ ┆ vities/13 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 480080256 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴───────────┴───────────┴───────────┘\n"
]
}
],
"source": [
"import polars as pl\n",
"import gpxpy\n",
"import numpy as np\n",
"\n",
"def gpx_to_polars(fname):\n",
" # Parse the GPX file\n",
" data = gpxpy.parse(open(fname))\n",
" prev = None\n",
" data_dict = {'course': [],\n",
" 'distance_2d': [],\n",
" 'latitude': [],\n",
" 'longitude': [],\n",
" 'time': [],\n",
" 'elevation': [],\n",
" 'speed_between': [],\n",
" }\n",
" name = None\n",
" type_ = None\n",
" filename = fname\n",
"\n",
" # Iterate through tracks, segments, and points\n",
" for track in data.tracks:\n",
" if name is None:\n",
" name = track.name\n",
" type_ = track.type\n",
"\n",
" for seg in track.segments:\n",
" for i, pt in enumerate(seg.points):\n",
" if prev is None:\n",
" prev = pt\n",
" for key in data_dict:\n",
" attr = getattr(pt, key)\n",
" if callable(attr):\n",
" data_dict[key].append(attr(prev))\n",
" else:\n",
" data_dict[key].append(attr)\n",
" prev = pt\n",
" data_dict['name'] = [name] * len(data_dict['course'])\n",
" data_dict['type'] = [type_] * len(data_dict['course'])\n",
" data_dict['filename'] = [filename] * len(data_dict['course'])\n",
" # Create a Polars DataFrame\n",
" df = (pl.DataFrame(data_dict)\n",
" )\n",
" return df\n",
" (df\n",
" #.with_columns([pl.col(\"time\").str.strptime(pl.Datetime, \"%Y-%m-%dT%H:%M:%SZ\", strict=False)])\n",
" .with_columns(\n",
" travelled= pl.col(\"distance_2d\").cum_sum(),\n",
" elapsed=(pl.col(\"time\") - pl.col(\"time\").min()).dt.total_seconds()\n",
" )\n",
" .with_columns(\n",
" avg_velocity=pl.col(\"travelled\") / pl.col(\"elapsed\"),\n",
" rolling_travelled=pl.col(\"travelled\").rolling_mean(window_size=5),\n",
" rolling_elapsed=pl.col(\"elapsed\").rolling_mean(window_size=5),\n",
" )\n",
" .with_columns(\n",
" rolling_velocity=pl.col(\"rolling_travelled\") / pl.col(\"rolling_elapsed\"),\n",
" rolling_between=pl.col(\"speed_between\").rolling_mean(window_size=5),\n",
" )\n",
" )\n",
"\n",
" return df\n",
"\n",
"# Example usage:\n",
"#crash_gpx = gpxpy.parse(open('/Users/matt/Downloads/Face_plant.gpx'))\n",
"df = gpx_to_polars('data/activities/13480080256.gpx')\n",
"print(df)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "TTpbSjcYb0QV"
},
"outputs": [],
"source": [
"import glob\n",
"\n",
"files = glob.glob('data/activities/*.gpx')\n",
"dfs = []\n",
"for file in files:\n",
" df = gpx_to_polars(file)\n",
" dfs.append(df)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "T2LvhVw0b0QV"
},
"outputs": [],
"source": [
"(pl.concat(dfs)\n",
" .write_parquet('data/activities.parquet')\n",
")\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "u9GfD8KVb0QV"
},
"outputs": [],
"source": [
"(#tweak_strava(raw)\n",
"raw\n",
"# make time naive - no tz\n",
" .with_columns(time=pl.col('time').dt.replace_time_zone(None))\n",
" .write_parquet('activities_naive.parquet')\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "BuNIhjerb0QV"
},
"source": [
"\n",
"## Load Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 654
},
"id": "PV0mJdKlb0QV",
"outputId": "e2056135-3a81-4ec0-fff8-ac2fed949724"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 10)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬───────────┬───────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ speed_bet ┆ name ┆ type ┆ filename │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ ween ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ --- ┆ str ┆ str ┆ str │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ f64 ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ null ┆ 365 and ┆ ebikeride ┆ data/acti │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ spork ┆ ┆ vities/11 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 6.704555 ┆ 41.370844 ┆ -111.90171 ┆ … ┆ 6.705301 ┆ 365 and ┆ ebikeride ┆ data/acti │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ spork ┆ ┆ vities/11 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 4.160126 ┆ 41.370807 ┆ -111.90171 ┆ … ┆ 4.161328 ┆ 365 and ┆ ebikeride ┆ data/acti │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ ┆ spork ┆ ┆ vities/11 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 2.62628 ┆ 41.370784 ┆ -111.90172 ┆ … ┆ 2.62628 ┆ 365 and ┆ ebikeride ┆ data/acti │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ spork ┆ ┆ vities/11 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ 1.273223 ┆ 365 and ┆ ebikeride ┆ data/acti │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ spork ┆ ┆ vities/11 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ 0.172201 ┆ Morning ┆ walking ┆ data/acti │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ walk ┆ ┆ vities/59 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx │\n",
"│ null ┆ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ … ┆ 0.238017 ┆ Morning ┆ walking ┆ data/acti │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ walk ┆ ┆ vities/59 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx │\n",
"│ null ┆ 1.05693 ┆ 40.879291 ┆ -111.85536 ┆ … ┆ 0.528465 ┆ Morning ┆ walking ┆ data/acti │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ walk ┆ ┆ vities/59 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx │\n",
"│ null ┆ 1.259495 ┆ 40.879301 ┆ -111.85537 ┆ … ┆ 0.421153 ┆ Morning ┆ walking ┆ data/acti │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ walk ┆ ┆ vities/59 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx │\n",
"│ null ┆ 1.023777 ┆ 40.879309 ┆ -111.85537 ┆ … ┆ 1.023777 ┆ Morning ┆ walking ┆ data/acti │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ walk ┆ ┆ vities/59 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴───────────┴───────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 10)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th></tr><tr><td>null</td><td>f64</td><td>f64</td><td>f64</td><td>datetime[μs, UTC]</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;16:54:56&nbsp;UTC</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>null</td><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>2024-06-29&nbsp;16:54:57&nbsp;UTC</td><td>1712.9</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>null</td><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>2024-06-29&nbsp;16:54:58&nbsp;UTC</td><td>1712.8</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>null</td><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>2024-06-29&nbsp;16:54:59&nbsp;UTC</td><td>1712.8</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>2024-06-29&nbsp;16:55:00&nbsp;UTC</td><td>1712.7</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>2021-09-09&nbsp;16:01:42&nbsp;UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>null</td><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>2021-09-09&nbsp;16:01:43&nbsp;UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>null</td><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>2021-09-09&nbsp;16:01:45&nbsp;UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>null</td><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>2021-09-09&nbsp;16:01:48&nbsp;UTC</td><td>1479.1</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>null</td><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>2021-09-09&nbsp;16:01:49&nbsp;UTC</td><td>1479.1</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 63
}
],
"source": [
"import polars as pl\n",
"import polars.selectors as cs\n",
"file = '/content/drive/MyDrive/Colab Notebooks/clients/nvidia/GTC2024/data/activities.parquet'\n",
"#file = 'activities_naive.parquet'\n",
"raw = pl.read_parquet(file)\n",
"raw"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c2zX6r_wb0QV"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fXbAws6gb0QV"
},
"source": [
"## Types"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6DgAImEJb0QV",
"outputId": "db5ddeed-51f1-40e9-aec5-ba21c797acc3"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"['course', 'distance_2d', 'latitude', 'longitude', 'time', 'elevation', 'speed_between', 'name', 'type', 'filename']\n"
]
}
],
"source": [
"print(raw.columns)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8Nt84aaQb0QV",
"outputId": "8ea76f73-ed37-4162-c5de-35c47d4e8a75"
},
"outputs": [
{
"data": {
"text/plain": [
"[Null,\n",
" Float64,\n",
" Float64,\n",
" Float64,\n",
" Datetime(time_unit='us', time_zone='UTC'),\n",
" Float64,\n",
" Float64,\n",
" String,\n",
" String,\n",
" String]"
]
},
"execution_count": 72,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cols = ['course', 'distance_2d', 'latitude', 'longitude', 'time', 'elevation',\n",
" 'speed_between', 'name', 'type', 'filename']\n",
"raw[cols].dtypes"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "1J9PPp-Sb0QV",
"outputId": "da197c68-a485-42dd-91f5-07509db9095b"
},
"outputs": [
{
"data": {
"text/plain": [
"506546445"
]
},
"execution_count": 91,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# 506 MB\n",
"raw[cols].estimated_size()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "XBwkc8Pzb0QV"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Jmf9uR_wb0QV"
},
"source": [
"## Ints"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qSAANvE2b0QV",
"outputId": "5da75ce1-0c83-44d6-fd2a-9e698256dfdb"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 11)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th></tr><tr><td>null</td><td>f64</td><td>f64</td><td>f64</td><td>datetime[μs, UTC]</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>i64</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>2024-06-29 16:54:57 UTC</td><td>1712.9</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>2024-06-29 16:54:58 UTC</td><td>1712.8</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.8</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>2024-06-29 16:55:00 UTC</td><td>1712.7</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>2021-09-09 16:01:48 UTC</td><td>1479.1</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.1</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 11)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬───────────┬───────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ name ┆ type ┆ filename ┆ id │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ str ┆ str ┆ str ┆ i64 │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370844 ┆ -111.90171 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370807 ┆ -111.90171 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370784 ┆ -111.90172 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879291 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879301 ┆ -111.85537 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879309 ┆ -111.85537 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴───────────┴───────────┴───────────┘"
]
},
"execution_count": 80,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# add a id from the filename\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "pzNvp7Dkb0QV",
"outputId": "1629c1a9-df6e-4eb8-ca04-4ea0a9f56661"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 11)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th></tr><tr><td>null</td><td>f64</td><td>f64</td><td>f64</td><td>datetime[μs, UTC]</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>i64</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>2024-06-29 16:54:57 UTC</td><td>1712.9</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>2024-06-29 16:54:58 UTC</td><td>1712.8</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.8</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>2024-06-29 16:55:00 UTC</td><td>1712.7</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>2021-09-09 16:01:48 UTC</td><td>1479.1</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr><tr><td>null</td><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.1</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 11)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬───────────┬───────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ name ┆ type ┆ filename ┆ id │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ str ┆ str ┆ str ┆ i64 │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370844 ┆ -111.90171 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370807 ┆ -111.90171 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370784 ┆ -111.90172 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ 365 and ┆ ebikeride ┆ data/acti ┆ 117698380 │\n",
"│ ┆ ┆ ┆ ┆ ┆ spork ┆ ┆ vities/11 ┆ 26 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 769838026 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ .gp… ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879291 ┆ -111.85536 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879301 ┆ -111.85537 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879309 ┆ -111.85537 ┆ … ┆ Morning ┆ walking ┆ data/acti ┆ 593262822 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ walk ┆ ┆ vities/59 ┆ 8 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ 32628228. ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ┆ gpx ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴───────────┴───────────┴───────────┘"
]
},
"execution_count": 84,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# add an elapsed column to each activity\n",
"# add a id from the filename\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id))\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "qtHmKREqb0QV",
"outputId": "79a79525-e05b-483a-ea47-3b337eddf976"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th></tr><tr><td>null</td><td>f64</td><td>f64</td><td>f64</td><td>datetime[μs, UTC]</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i64</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td></tr><tr><td>null</td><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>2024-06-29 16:54:57 UTC</td><td>1712.9</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td></tr><tr><td>null</td><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>2024-06-29 16:54:58 UTC</td><td>1712.8</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td></tr><tr><td>null</td><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.8</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>2024-06-29 16:55:00 UTC</td><td>1712.7</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td></tr><tr><td>null</td><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td></tr><tr><td>null</td><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td></tr><tr><td>null</td><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>2021-09-09 16:01:48 UTC</td><td>1479.1</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td></tr><tr><td>null</td><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.1</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬────────────┬────────────┬─────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ type ┆ filename ┆ id ┆ elapsed │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ str ┆ str ┆ i64 ┆ i64 │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪════════════╪════════════╪═════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 0 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370844 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 1 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370807 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 2 │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370784 ┆ -111.90172 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 3 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 4 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4275 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4276 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879291 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4278 │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879301 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4281 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879309 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4282 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴────────────┴────────────┴─────────┘"
]
},
"execution_count": 87,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# add an elapsed column to each activity\n",
"# add a id from the filename\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed)\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "vuxd-45ab0QW",
"outputId": "071c9c05-03d3-48ec-cb1d-73dea42b6efe"
},
"outputs": [
{
"ename": "InvalidOperationError",
"evalue": "conversion from `i64` to `i16` failed in column 'time' for 231842 out of 5050370 values: [32768, 32769, … 82899]",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mInvalidOperationError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[88], line 15\u001b[0m\n\u001b[1;32m 3\u001b[0m calc_id \u001b[38;5;241m=\u001b[39m (pl\u001b[38;5;241m.\u001b[39mcol(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mfilename\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 4\u001b[0m \u001b[38;5;241m.\u001b[39mstr\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m/\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 5\u001b[0m \u001b[38;5;241m.\u001b[39mlist\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m)\n\u001b[1;32m 6\u001b[0m \u001b[38;5;241m.\u001b[39mstr\u001b[38;5;241m.\u001b[39mreplace(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m.gpx\u001b[39m\u001b[38;5;124m'\u001b[39m, \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 7\u001b[0m \u001b[38;5;241m.\u001b[39mcast(pl\u001b[38;5;241m.\u001b[39mInt64)\n\u001b[1;32m 8\u001b[0m )\n\u001b[1;32m 9\u001b[0m calc_elapsed \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m 10\u001b[0m (pl\u001b[38;5;241m.\u001b[39mcol(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtime\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;241m-\u001b[39m pl\u001b[38;5;241m.\u001b[39mcol(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtime\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mmin())\n\u001b[1;32m 11\u001b[0m \u001b[38;5;241m.\u001b[39mdt\u001b[38;5;241m.\u001b[39mtotal_seconds()\n\u001b[1;32m 12\u001b[0m \u001b[38;5;241m.\u001b[39mover(pl\u001b[38;5;241m.\u001b[39mcol(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mid\u001b[39m\u001b[38;5;124m'\u001b[39m))\n\u001b[1;32m 13\u001b[0m )\n\u001b[0;32m---> 15\u001b[0m (\u001b[43mraw\u001b[49m\n\u001b[1;32m 16\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mselect\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcols\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 17\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;66;43;03m# add elapsed time\u001b[39;49;00m\n\u001b[1;32m 18\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwith_columns\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mid\u001b[39;49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcalc_id\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 19\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;66;43;03m# needs to be in its own with_columns because we reference id\u001b[39;49;00m\n\u001b[1;32m 20\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwith_columns\u001b[49m\u001b[43m(\u001b[49m\u001b[43melapsed\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcalc_elapsed\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcast\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpl\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mInt16\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 21\u001b[0m \n\u001b[1;32m 22\u001b[0m )\n",
"File \u001b[0;32m~/.envs/menv/lib/python3.10/site-packages/polars/dataframe/frame.py:9586\u001b[0m, in \u001b[0;36mDataFrame.with_columns\u001b[0;34m(self, *exprs, **named_exprs)\u001b[0m\n\u001b[1;32m 9440\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21mwith_columns\u001b[39m(\n\u001b[1;32m 9441\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 9442\u001b[0m \u001b[38;5;241m*\u001b[39mexprs: IntoExpr \u001b[38;5;241m|\u001b[39m Iterable[IntoExpr],\n\u001b[1;32m 9443\u001b[0m \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mnamed_exprs: IntoExpr,\n\u001b[1;32m 9444\u001b[0m ) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m DataFrame:\n\u001b[1;32m 9445\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m 9446\u001b[0m \u001b[38;5;124;03m Add columns to this DataFrame.\u001b[39;00m\n\u001b[1;32m 9447\u001b[0m \n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 9584\u001b[0m \u001b[38;5;124;03m └─────┴──────┴─────────────┘\u001b[39;00m\n\u001b[1;32m 9585\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[0;32m-> 9586\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mlazy\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mwith_columns\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mexprs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mnamed_exprs\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcollect\u001b[49m\u001b[43m(\u001b[49m\u001b[43m_eager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\n",
"File \u001b[0;32m~/.envs/menv/lib/python3.10/site-packages/polars/lazyframe/frame.py:2056\u001b[0m, in \u001b[0;36mLazyFrame.collect\u001b[0;34m(self, type_coercion, _type_check, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, streaming, engine, background, _check_order, _eager, **_kwargs)\u001b[0m\n\u001b[1;32m 2054\u001b[0m \u001b[38;5;66;03m# Only for testing purposes\u001b[39;00m\n\u001b[1;32m 2055\u001b[0m callback \u001b[38;5;241m=\u001b[39m _kwargs\u001b[38;5;241m.\u001b[39mget(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mpost_opt_callback\u001b[39m\u001b[38;5;124m\"\u001b[39m, callback)\n\u001b[0;32m-> 2056\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m wrap_df(\u001b[43mldf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcollect\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcallback\u001b[49m\u001b[43m)\u001b[49m)\n",
"\u001b[0;31mInvalidOperationError\u001b[0m: conversion from `i64` to `i16` failed in column 'time' for 231842 out of 5050370 values: [32768, 32769, … 82899]"
]
}
],
"source": [
"# cast ints to int16\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int16))\n",
"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ik7-wJ0nb0QW",
"outputId": "3e10a354-6052-461d-dd29-79a3dac97222"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>statistic</th><th>id</th><th>elapsed</th></tr><tr><td>str</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;count&quot;</td><td>5.05037e6</td><td>5.05037e6</td></tr><tr><td>&quot;null_count&quot;</td><td>0.0</td><td>0.0</td></tr><tr><td>&quot;mean&quot;</td><td>7.3017e9</td><td>289710.218237</td></tr><tr><td>&quot;std&quot;</td><td>3.8561e9</td><td>1.8556e7</td></tr><tr><td>&quot;min&quot;</td><td>4.86569874e8</td><td>0.0</td></tr><tr><td>&quot;25%&quot;</td><td>3.8631e9</td><td>1843.0</td></tr><tr><td>&quot;50%&quot;</td><td>8.0096e9</td><td>4121.0</td></tr><tr><td>&quot;75%&quot;</td><td>1.0564e10</td><td>8384.0</td></tr><tr><td>&quot;max&quot;</td><td>1.3480e10</td><td>1.2386e9</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (9, 3)\n",
"┌────────────┬──────────────┬───────────────┐\n",
"│ statistic ┆ id ┆ elapsed │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ str ┆ f64 ┆ f64 │\n",
"╞════════════╪══════════════╪═══════════════╡\n",
"│ count ┆ 5.05037e6 ┆ 5.05037e6 │\n",
"│ null_count ┆ 0.0 ┆ 0.0 │\n",
"│ mean ┆ 7.3017e9 ┆ 289710.218237 │\n",
"│ std ┆ 3.8561e9 ┆ 1.8556e7 │\n",
"│ min ┆ 4.86569874e8 ┆ 0.0 │\n",
"│ 25% ┆ 3.8631e9 ┆ 1843.0 │\n",
"│ 50% ┆ 8.0096e9 ┆ 4121.0 │\n",
"│ 75% ┆ 1.0564e10 ┆ 8384.0 │\n",
"│ max ┆ 1.3480e10 ┆ 1.2386e9 │\n",
"└────────────┴──────────────┴───────────────┘"
]
},
"execution_count": 95,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# describe\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed)\n",
" .select(cs.integer())\n",
" .describe()\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "55mMPVx6b0QW",
"outputId": "952a07e6-ea48-4a81-aea2-efdd3f448f11"
},
"outputs": [
{
"data": {
"text/plain": [
"iinfo(min=-32768, max=32767, dtype=int16)"
]
},
"execution_count": 96,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"np.iinfo(np.int16)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6LvtSwg1b0QW",
"outputId": "dddc88d4-39e0-427b-f3d0-c101be53f20d"
},
"outputs": [
{
"data": {
"text/plain": [
"iinfo(min=-2147483648, max=2147483647, dtype=int32)"
]
},
"execution_count": 97,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import numpy as np\n",
"np.iinfo(np.int32)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WAmNveytb0QW",
"outputId": "889fad07-d58d-4d9c-b45e-5f1df78879de"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th></tr><tr><td>null</td><td>f64</td><td>f64</td><td>f64</td><td>datetime[μs, UTC]</td><td>f64</td><td>f64</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i32</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td></tr><tr><td>null</td><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>2024-06-29 16:54:57 UTC</td><td>1712.9</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td></tr><tr><td>null</td><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>2024-06-29 16:54:58 UTC</td><td>1712.8</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td></tr><tr><td>null</td><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.8</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>2024-06-29 16:55:00 UTC</td><td>1712.7</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td></tr><tr><td>null</td><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td></tr><tr><td>null</td><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td></tr><tr><td>null</td><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>2021-09-09 16:01:48 UTC</td><td>1479.1</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td></tr><tr><td>null</td><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.1</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬────────────┬────────────┬─────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ type ┆ filename ┆ id ┆ elapsed │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f64 ┆ f64 ┆ ┆ str ┆ str ┆ i64 ┆ i32 │\n",
"│ ┆ f64 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪════════════╪════════════╪═════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 0 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370844 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 1 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370807 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 2 │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370784 ┆ -111.90172 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 3 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 4 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4275 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4276 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879291 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4278 │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879301 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4281 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879309 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4282 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴────────────┴────────────┴─────────┘"
]
},
"execution_count": 89,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast ints to int16\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "CC1LDWhhb0QW",
"outputId": "cd6ac1f9-89e7-4e5b-f320-653efb2875c7"
},
"outputs": [
{
"data": {
"text/plain": [
"567150885"
]
},
"execution_count": 90,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast ints to int16\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .estimated_size()\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "exA3RvO5b0QW"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "HVnVCbiob0QW"
},
"source": [
"## Floats\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "l2GDDsIub0QW",
"outputId": "8df59bad-c2f8-4732-c0be-3b13e4ec4319"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 5)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>elevation</th><th>speed_between</th></tr><tr><td>f64</td><td>f64</td><td>f64</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>1713.0</td><td>null</td></tr><tr><td>6.704555</td><td>41.370844</td><td>-111.901712</td><td>1712.9</td><td>6.705301</td></tr><tr><td>4.160126</td><td>41.370807</td><td>-111.901719</td><td>1712.8</td><td>4.161328</td></tr><tr><td>2.62628</td><td>41.370784</td><td>-111.901726</td><td>1712.8</td><td>2.62628</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.90173</td><td>1712.7</td><td>1.273223</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855361</td><td>1479.0</td><td>0.172201</td></tr><tr><td>0.238017</td><td>40.879282</td><td>-111.85536</td><td>1479.0</td><td>0.238017</td></tr><tr><td>1.05693</td><td>40.879291</td><td>-111.855364</td><td>1479.0</td><td>0.528465</td></tr><tr><td>1.259495</td><td>40.879301</td><td>-111.855371</td><td>1479.1</td><td>0.421153</td></tr><tr><td>1.023777</td><td>40.879309</td><td>-111.855377</td><td>1479.1</td><td>1.023777</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 5)\n",
"┌─────────────┬───────────┬─────────────┬───────────┬───────────────┐\n",
"│ distance_2d ┆ latitude ┆ longitude ┆ elevation ┆ speed_between │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │\n",
"╞═════════════╪═══════════╪═════════════╪═══════════╪═══════════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.901772 ┆ 1713.0 ┆ null │\n",
"│ 6.704555 ┆ 41.370844 ┆ -111.901712 ┆ 1712.9 ┆ 6.705301 │\n",
"│ 4.160126 ┆ 41.370807 ┆ -111.901719 ┆ 1712.8 ┆ 4.161328 │\n",
"│ 2.62628 ┆ 41.370784 ┆ -111.901726 ┆ 1712.8 ┆ 2.62628 │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ 1712.7 ┆ 1.273223 │\n",
"│ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.855361 ┆ 1479.0 ┆ 0.172201 │\n",
"│ 0.238017 ┆ 40.879282 ┆ -111.85536 ┆ 1479.0 ┆ 0.238017 │\n",
"│ 1.05693 ┆ 40.879291 ┆ -111.855364 ┆ 1479.0 ┆ 0.528465 │\n",
"│ 1.259495 ┆ 40.879301 ┆ -111.855371 ┆ 1479.1 ┆ 0.421153 │\n",
"│ 1.023777 ┆ 40.879309 ┆ -111.855377 ┆ 1479.1 ┆ 1.023777 │\n",
"└─────────────┴───────────┴─────────────┴───────────┴───────────────┘"
]
},
"execution_count": 98,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .select(cs.float())\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Z1Ehq7_8b0QW",
"outputId": "8000e794-bb69-4912-8e53-21159ff1db8d"
},
"outputs": [
{
"data": {
"text/plain": [
"['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']"
]
},
"execution_count": 112,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .select(cs.float())\n",
" .columns\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "x03Lx0w4b0QW",
"outputId": "ab45299b-63cf-4ce9-a97a-87702d4fa8d5"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th></tr><tr><td>null</td><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, UTC]</td><td>f32</td><td>f32</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i32</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td></tr><tr><td>null</td><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29 16:54:57 UTC</td><td>1712.900024</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td></tr><tr><td>null</td><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29 16:54:58 UTC</td><td>1712.800049</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td></tr><tr><td>null</td><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.800049</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29 16:55:00 UTC</td><td>1712.699951</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td></tr><tr><td>null</td><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td></tr><tr><td>null</td><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td></tr><tr><td>null</td><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09 16:01:48 UTC</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td></tr><tr><td>null</td><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬────────────┬────────────┬─────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ type ┆ filename ┆ id ┆ elapsed │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f32 ┆ f32 ┆ ┆ str ┆ str ┆ i64 ┆ i32 │\n",
"│ ┆ f32 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪════════════╪════════════╪═════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 0 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370846 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 1 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370808 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 2 │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370785 ┆ -111.90172 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 3 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 4 │\n",
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4275 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879284 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4276 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879292 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4278 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879299 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4281 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879311 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4282 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴────────────┴────────────┴─────────┘"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# need less than 7 decimal digits of precision\n",
"# convert to float32\n",
"# cast\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .with_columns(pl.col(float_cols).cast(pl.Float32))\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "itp8eWSUb0QX"
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Jm8eho6eb0QX"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "vFCFhOR5b0QX"
},
"source": [
"## Strings"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "cQr4EYIib0QX",
"outputId": "98bddf27-eddf-4c20-852a-9a4644821a61"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>name</th><th>type</th><th>filename</th></tr><tr><td>str</td><td>str</td><td>str</td></tr></thead><tbody><tr><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr><tr><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 3)\n",
"┌───────────────┬───────────┬─────────────────────────────────┐\n",
"│ name ┆ type ┆ filename │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ str ┆ str ┆ str │\n",
"╞═══════════════╪═══════════╪═════════════════════════════════╡\n",
"│ 365 and spork ┆ ebikeride ┆ data/activities/11769838026.gp… │\n",
"│ 365 and spork ┆ ebikeride ┆ data/activities/11769838026.gp… │\n",
"│ 365 and spork ┆ ebikeride ┆ data/activities/11769838026.gp… │\n",
"│ 365 and spork ┆ ebikeride ┆ data/activities/11769838026.gp… │\n",
"│ 365 and spork ┆ ebikeride ┆ data/activities/11769838026.gp… │\n",
"│ … ┆ … ┆ … │\n",
"│ Morning walk ┆ walking ┆ data/activities/5932628228.gpx │\n",
"│ Morning walk ┆ walking ┆ data/activities/5932628228.gpx │\n",
"│ Morning walk ┆ walking ┆ data/activities/5932628228.gpx │\n",
"│ Morning walk ┆ walking ┆ data/activities/5932628228.gpx │\n",
"│ Morning walk ┆ walking ┆ data/activities/5932628228.gpx │\n",
"└───────────────┴───────────┴─────────────────────────────────┘"
]
},
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .with_columns(pl.col(float_cols).cast(pl.Float32))\n",
" .select(cs.string())\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WWcuhXrPb0Qa",
"outputId": "1e08395e-9a2b-4681-94de-4bebac6dfcea"
},
"outputs": [
{
"data": {
"text/plain": [
"<polars.expr.string.ExprStringNameSpace at 0x14699f070>"
]
},
"execution_count": 120,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pl.col('').str"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dr_YCcOdb0Qa",
"outputId": "4f3cb3c0-0ef0-447c-e187-6682eb7dfd31"
},
"outputs": [
{
"data": {
"text/plain": [
"['__class__',\n",
" '__delattr__',\n",
" '__dict__',\n",
" '__dir__',\n",
" '__doc__',\n",
" '__eq__',\n",
" '__format__',\n",
" '__ge__',\n",
" '__getattribute__',\n",
" '__gt__',\n",
" '__hash__',\n",
" '__init__',\n",
" '__init_subclass__',\n",
" '__le__',\n",
" '__lt__',\n",
" '__module__',\n",
" '__ne__',\n",
" '__new__',\n",
" '__reduce__',\n",
" '__reduce_ex__',\n",
" '__repr__',\n",
" '__setattr__',\n",
" '__sizeof__',\n",
" '__str__',\n",
" '__subclasshook__',\n",
" '__weakref__',\n",
" '_accessor',\n",
" '_pyexpr',\n",
" 'concat',\n",
" 'contains',\n",
" 'contains_any',\n",
" 'count_matches',\n",
" 'decode',\n",
" 'encode',\n",
" 'ends_with',\n",
" 'escape_regex',\n",
" 'explode',\n",
" 'extract',\n",
" 'extract_all',\n",
" 'extract_groups',\n",
" 'extract_many',\n",
" 'find',\n",
" 'find_many',\n",
" 'head',\n",
" 'join',\n",
" 'json_decode',\n",
" 'json_path_match',\n",
" 'len_bytes',\n",
" 'len_chars',\n",
" 'normalize',\n",
" 'pad_end',\n",
" 'pad_start',\n",
" 'replace',\n",
" 'replace_all',\n",
" 'replace_many',\n",
" 'reverse',\n",
" 'slice',\n",
" 'split',\n",
" 'split_exact',\n",
" 'splitn',\n",
" 'starts_with',\n",
" 'strip_chars',\n",
" 'strip_chars_end',\n",
" 'strip_chars_start',\n",
" 'strip_prefix',\n",
" 'strip_suffix',\n",
" 'strptime',\n",
" 'tail',\n",
" 'to_date',\n",
" 'to_datetime',\n",
" 'to_decimal',\n",
" 'to_integer',\n",
" 'to_lowercase',\n",
" 'to_time',\n",
" 'to_titlecase',\n",
" 'to_uppercase',\n",
" 'zfill']"
]
},
"execution_count": 121,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dir(pl.col('').str)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "KOkpOSQXb0Qa",
"outputId": "b23f621e-46cb-4b15-bcb0-4f666392bad0"
},
"outputs": [
{
"data": {
"text/plain": [
"263290777"
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast to categorical removes 50% of the size\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .with_columns(pl.col(float_cols).cast(pl.Float32))\n",
" .with_columns(cs.string().cast(pl.Categorical))\n",
" .estimated_size()\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VD3y1LpTb0Qa",
"outputId": "01629417-a59a-4d5e-d729-2d2c3860bd56"
},
"outputs": [
{
"data": {
"text/plain": [
"['name', 'type', 'filename']"
]
},
"execution_count": 125,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast to categorical removes 50% of the size\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(elapsed=calc_elapsed.cast(pl.Int32))\n",
" .with_columns(pl.col(float_cols).cast(pl.Float32))\n",
" #.with_columns(cs.string().cast(pl.Categorical))\n",
" .select(cs.string())\n",
" .columns\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "6L4WvjaQb0Qa",
"outputId": "05cd54e9-02fe-4473-ebce-5c39e51934cc"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th></tr><tr><td>null</td><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, UTC]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 16:54:56 UTC</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td></tr><tr><td>null</td><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29 16:54:57 UTC</td><td>1712.900024</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td></tr><tr><td>null</td><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29 16:54:58 UTC</td><td>1712.800049</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td></tr><tr><td>null</td><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29 16:54:59 UTC</td><td>1712.800049</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29 16:55:00 UTC</td><td>1712.699951</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09 16:01:42 UTC</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td></tr><tr><td>null</td><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09 16:01:43 UTC</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td></tr><tr><td>null</td><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09 16:01:45 UTC</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td></tr><tr><td>null</td><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09 16:01:48 UTC</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td></tr><tr><td>null</td><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09 16:01:49 UTC</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬────────────┬────────────┬─────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ type ┆ filename ┆ id ┆ elapsed │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f32 ┆ f32 ┆ ┆ cat ┆ cat ┆ i64 ┆ i32 │\n",
"│ ┆ f32 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪════════════╪════════════╪═════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 0 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370846 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 1 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370808 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 2 │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370785 ┆ -111.90172 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 3 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 4 │\n",
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4275 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879284 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4276 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879292 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4278 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879299 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4281 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879311 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4282 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴────────────┴────────────┴─────────┘"
]
},
"execution_count": 127,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# cast to categorical removes 50% of the size\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "gEnicOApb0Qa",
"outputId": "4b0b79fc-fb2e-48ef-b4d9-89dc1cfb1f94"
},
"outputs": [
{
"data": {
"text/plain": [
"<polars.expr.categorical.ExprCatNameSpace at 0x3247797b0>"
]
},
"execution_count": 119,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pl.col('cat').cat"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "_g4SYzJMb0Qa",
"outputId": "616d3fe4-7aa8-45dc-9d91-6f56a030852d"
},
"outputs": [
{
"data": {
"text/plain": [
"['__class__',\n",
" '__delattr__',\n",
" '__dict__',\n",
" '__dir__',\n",
" '__doc__',\n",
" '__eq__',\n",
" '__format__',\n",
" '__ge__',\n",
" '__getattribute__',\n",
" '__gt__',\n",
" '__hash__',\n",
" '__init__',\n",
" '__init_subclass__',\n",
" '__le__',\n",
" '__lt__',\n",
" '__module__',\n",
" '__ne__',\n",
" '__new__',\n",
" '__reduce__',\n",
" '__reduce_ex__',\n",
" '__repr__',\n",
" '__setattr__',\n",
" '__sizeof__',\n",
" '__str__',\n",
" '__subclasshook__',\n",
" '__weakref__',\n",
" '_accessor',\n",
" '_pyexpr',\n",
" 'ends_with',\n",
" 'get_categories',\n",
" 'len_bytes',\n",
" 'len_chars',\n",
" 'starts_with']"
]
},
"execution_count": 122,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dir(pl.col('cat').cat)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "kCuYx-7Yb0Qa"
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "ba3TaTSSb0Qa"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "I1B7RzXsb0Qa"
},
"source": [
"## Dates\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "3ByN8k-vb0Qa",
"outputId": "a49eb9c1-35b7-4dbe-d120-82c9751f805f"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th></tr><tr><td>null</td><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 10:54:56 MDT</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td></tr><tr><td>null</td><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29 10:54:57 MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td></tr><tr><td>null</td><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29 10:54:58 MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td></tr><tr><td>null</td><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29 10:54:59 MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29 10:55:00 MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09 10:01:42 MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td></tr><tr><td>null</td><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09 10:01:43 MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td></tr><tr><td>null</td><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09 10:01:45 MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td></tr><tr><td>null</td><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09 10:01:48 MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td></tr><tr><td>null</td><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09 10:01:49 MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬───────────┬────────────┬────────────┬─────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ type ┆ filename ┆ id ┆ elapsed │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f32 ┆ f32 ┆ ┆ cat ┆ cat ┆ i64 ┆ i32 │\n",
"│ ┆ f32 ┆ ┆ ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪═══════════╪════════════╪════════════╪═════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 0 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370846 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 1 │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370808 ┆ -111.90171 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 2 │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370785 ┆ -111.90172 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 3 │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ ebikeride ┆ data/activ ┆ 1176983802 ┆ 4 │\n",
"│ ┆ ┆ ┆ 3 ┆ ┆ ┆ ities/1176 ┆ 6 ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4275 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879284 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4276 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879292 ┆ -111.85536 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4278 │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879299 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4281 │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879311 ┆ -111.85537 ┆ … ┆ walking ┆ data/activ ┆ 5932628228 ┆ 4282 │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ┆ ities/5932 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴───────────┴────────────┴────────────┴─────────┘"
]
},
"execution_count": 129,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# convert date to local time\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8VtnCyIBb0Qa",
"outputId": "c8dc2dfe-f739-4359-8a28-ed5da46d201c"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>null</td><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>duration[μs]</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 10:54:56 MDT</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29 10:54:57 MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td></tr><tr><td>null</td><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29 10:54:58 MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td></tr><tr><td>null</td><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29 10:54:59 MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td></tr><tr><td>null</td><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29 10:55:00 MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09 10:01:42 MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h 11m 15s</td></tr><tr><td>null</td><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09 10:01:43 MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h 11m 16s</td></tr><tr><td>null</td><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09 10:01:45 MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h 11m 18s</td></tr><tr><td>null</td><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09 10:01:48 MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h 11m 21s</td></tr><tr><td>null</td><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09 10:01:49 MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h 11m 22s</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 13)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬────────────┬────────────┬─────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f32 ┆ f32 ┆ ┆ cat ┆ i64 ┆ i32 ┆ duration[ │\n",
"│ ┆ f32 ┆ ┆ ┆ ┆ ┆ ┆ ┆ μs] │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪════════════╪════════════╪═════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ data/activ ┆ 1176983802 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ null ┆ 6.704555 ┆ 41.370846 ┆ -111.90171 ┆ … ┆ data/activ ┆ 1176983802 ┆ 1 ┆ 1s │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ null ┆ 4.160126 ┆ 41.370808 ┆ -111.90171 ┆ … ┆ data/activ ┆ 1176983802 ┆ 2 ┆ 2s │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ null ┆ 2.62628 ┆ 41.370785 ┆ -111.90172 ┆ … ┆ data/activ ┆ 1176983802 ┆ 3 ┆ 3s │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ null ┆ 1.26929 ┆ 41.370773 ┆ -111.90173 ┆ … ┆ data/activ ┆ 1176983802 ┆ 4 ┆ 4s │\n",
"│ ┆ ┆ ┆ 3 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 0.344402 ┆ 40.87928 ┆ -111.85536 ┆ … ┆ data/activ ┆ 5932628228 ┆ 4275 ┆ 1h 11m │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/5932 ┆ ┆ ┆ 15s │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.238017 ┆ 40.879284 ┆ -111.85536 ┆ … ┆ data/activ ┆ 5932628228 ┆ 4276 ┆ 1h 11m │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/5932 ┆ ┆ ┆ 16s │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 1.05693 ┆ 40.879292 ┆ -111.85536 ┆ … ┆ data/activ ┆ 5932628228 ┆ 4278 ┆ 1h 11m │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/5932 ┆ ┆ ┆ 18s │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 1.259495 ┆ 40.879299 ┆ -111.85537 ┆ … ┆ data/activ ┆ 5932628228 ┆ 4281 ┆ 1h 11m │\n",
"│ ┆ ┆ ┆ ┆ ┆ ities/5932 ┆ ┆ ┆ 21s │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 1.023777 ┆ 40.879311 ┆ -111.85537 ┆ … ┆ data/activ ┆ 5932628228 ┆ 4282 ┆ 1h 11m │\n",
"│ ┆ ┆ ┆ 7 ┆ ┆ ities/5932 ┆ ┆ ┆ 22s │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴────────────┴────────────┴─────────┴───────────┘"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# create elapsed time\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" .with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "0ocyHfjFb0Qa",
"outputId": "3c800f23-3876-4d84-8281-89c34dbe001f"
},
"outputs": [
{
"data": {
"text/plain": [
"<polars.expr.datetime.ExprDateTimeNameSpace at 0x14699f160>"
]
},
"execution_count": 130,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pl.col('time').dt"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "apueg-6sb0Qa",
"outputId": "c4792218-a89a-4682-c6e3-81ed130efc4e"
},
"outputs": [
{
"data": {
"text/plain": [
"['__class__',\n",
" '__delattr__',\n",
" '__dict__',\n",
" '__dir__',\n",
" '__doc__',\n",
" '__eq__',\n",
" '__format__',\n",
" '__ge__',\n",
" '__getattribute__',\n",
" '__gt__',\n",
" '__hash__',\n",
" '__init__',\n",
" '__init_subclass__',\n",
" '__le__',\n",
" '__lt__',\n",
" '__module__',\n",
" '__ne__',\n",
" '__new__',\n",
" '__reduce__',\n",
" '__reduce_ex__',\n",
" '__repr__',\n",
" '__setattr__',\n",
" '__sizeof__',\n",
" '__str__',\n",
" '__subclasshook__',\n",
" '__weakref__',\n",
" '_accessor',\n",
" '_pyexpr',\n",
" 'add_business_days',\n",
" 'base_utc_offset',\n",
" 'cast_time_unit',\n",
" 'century',\n",
" 'combine',\n",
" 'convert_time_zone',\n",
" 'date',\n",
" 'datetime',\n",
" 'day',\n",
" 'dst_offset',\n",
" 'epoch',\n",
" 'hour',\n",
" 'is_leap_year',\n",
" 'iso_year',\n",
" 'microsecond',\n",
" 'millennium',\n",
" 'millisecond',\n",
" 'minute',\n",
" 'month',\n",
" 'month_end',\n",
" 'month_start',\n",
" 'nanosecond',\n",
" 'offset_by',\n",
" 'ordinal_day',\n",
" 'quarter',\n",
" 'replace',\n",
" 'replace_time_zone',\n",
" 'round',\n",
" 'second',\n",
" 'strftime',\n",
" 'time',\n",
" 'timestamp',\n",
" 'to_string',\n",
" 'total_days',\n",
" 'total_hours',\n",
" 'total_microseconds',\n",
" 'total_milliseconds',\n",
" 'total_minutes',\n",
" 'total_nanoseconds',\n",
" 'total_seconds',\n",
" 'truncate',\n",
" 'week',\n",
" 'weekday',\n",
" 'with_time_unit',\n",
" 'year']"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"dir(pl.col('time').dt)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "InPcWdv4b0Qb"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "fWxP_JNNb0Qb"
},
"source": [
"## Missing Data"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Ix8WRnVvb0Qb",
"outputId": "7c288f71-4ce6-4080-d3ed-960f22024c3f"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (1, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>5050370</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>2293</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (1, 13)\n",
"┌─────────┬─────────────┬──────────┬───────────┬───┬──────────┬─────┬─────────┬────────┐\n",
"│ course ┆ distance_2d ┆ latitude ┆ longitude ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ u32 ┆ u32 ┆ u32 ┆ u32 ┆ ┆ u32 ┆ u32 ┆ u32 ┆ u32 │\n",
"╞═════════╪═════════════╪══════════╪═══════════╪═══╪══════════╪═════╪═════════╪════════╡\n",
"│ 5050370 ┆ 0 ┆ 0 ┆ 0 ┆ … ┆ 0 ┆ 0 ┆ 0 ┆ 0 │\n",
"└─────────┴─────────────┴──────────┴───────────┴───┴──────────┴─────┴─────────┴────────┘"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# where are values missing (null)?\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" .with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
" .null_count()\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "yEwNZJ9Zb0Qb",
"outputId": "e36f718f-95c2-44b6-fc23-6ac6469b332e"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (2_293, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>course</th><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>null</td><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>duration[μs]</td></tr></thead><tbody><tr><td>null</td><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 10:54:56 MDT</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.0</td><td>38.651745</td><td>-109.668266</td><td>2021-11-26 13:03:10 MST</td><td>1392.300049</td><td>null</td><td>&quot;Bar m&quot;</td><td>&quot;cycling&quot;</td><td>&quot;data/activities/6310856608.gpx&quot;</td><td>6310856608</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.0</td><td>40.863651</td><td>-111.833046</td><td>2020-01-01 13:13:21 MST</td><td>1628.199951</td><td>null</td><td>&quot;Kenny Creek&quot;</td><td>&quot;Snowshoe&quot;</td><td>&quot;data/activities/2974143712.gpx&quot;</td><td>2974143712</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.276011</td><td>40.86248</td><td>-111.821152</td><td>2020-01-01 14:11:43 MST</td><td>1829.300049</td><td>null</td><td>&quot;Kenny Creek&quot;</td><td>&quot;Snowshoe&quot;</td><td>&quot;data/activities/2974143712.gpx&quot;</td><td>2974143712</td><td>3502</td><td>58m 22s</td></tr><tr><td>null</td><td>0.0</td><td>40.86248</td><td>-111.821152</td><td>2020-01-01 14:11:44 MST</td><td>1829.300049</td><td>null</td><td>&quot;Kenny Creek&quot;</td><td>&quot;Snowshoe&quot;</td><td>&quot;data/activities/2974143712.gpx&quot;</td><td>2974143712</td><td>3503</td><td>58m 23s</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>null</td><td>1.030039</td><td>40.851273</td><td>-111.823189</td><td>2020-09-21 09:42:31 MDT</td><td>1844.400024</td><td>null</td><td>&quot;Birthday hike/run&quot;</td><td>&quot;hiking&quot;</td><td>&quot;data/activities/4092734339.gpx&quot;</td><td>4092734339</td><td>3056</td><td>50m 56s</td></tr><tr><td>null</td><td>0.0</td><td>40.747234</td><td>-111.562019</td><td>2021-07-08 17:58:54 MDT</td><td>1918.599976</td><td>null</td><td>&quot;The dropout&quot;</td><td>&quot;cycling&quot;</td><td>&quot;data/activities/5597638888.gpx&quot;</td><td>5597638888</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.0</td><td>40.916195</td><td>-111.861771</td><td>2016-09-05 11:40:12 MDT</td><td>1449.300049</td><td>null</td><td>&quot;Deuel Creek Lunch Run&quot;</td><td>&quot;running&quot;</td><td>&quot;data/activities/701803795.gpx&quot;</td><td>701803795</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.0</td><td>46.914665</td><td>-121.640488</td><td>2022-07-15 13:54:52 MDT</td><td>1944.699951</td><td>null</td><td>&quot;Silver forest &quot;</td><td>&quot;hiking&quot;</td><td>&quot;data/activities/7475182621.gpx&quot;</td><td>7475182621</td><td>0</td><td>0µs</td></tr><tr><td>null</td><td>0.0</td><td>40.87928</td><td>-111.855194</td><td>2021-09-09 08:50:27 MDT</td><td>1479.099976</td><td>null</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>0</td><td>0µs</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (2_293, 13)\n",
"┌────────┬────────────┬───────────┬────────────┬───┬────────────┬────────────┬─────────┬───────────┐\n",
"│ course ┆ distance_2 ┆ latitude ┆ longitude ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ --- ┆ d ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ null ┆ --- ┆ f32 ┆ f32 ┆ ┆ cat ┆ i64 ┆ i32 ┆ duration[ │\n",
"│ ┆ f32 ┆ ┆ ┆ ┆ ┆ ┆ ┆ μs] │\n",
"╞════════╪════════════╪═══════════╪════════════╪═══╪════════════╪════════════╪═════════╪═══════════╡\n",
"│ null ┆ 0.0 ┆ 41.370804 ┆ -111.90177 ┆ … ┆ data/activ ┆ 1176983802 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/1176 ┆ 6 ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 9838026.gp ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ … ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 38.651745 ┆ -109.66826 ┆ … ┆ data/activ ┆ 6310856608 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ities/6310 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 856608.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 40.863651 ┆ -111.83304 ┆ … ┆ data/activ ┆ 2974143712 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 6 ┆ ┆ ities/2974 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 143712.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.276011 ┆ 40.86248 ┆ -111.82115 ┆ … ┆ data/activ ┆ 2974143712 ┆ 3502 ┆ 58m 22s │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/2974 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 143712.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 40.86248 ┆ -111.82115 ┆ … ┆ data/activ ┆ 2974143712 ┆ 3503 ┆ 58m 23s │\n",
"│ ┆ ┆ ┆ 2 ┆ ┆ ities/2974 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 143712.gpx ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ null ┆ 1.030039 ┆ 40.851273 ┆ -111.82318 ┆ … ┆ data/activ ┆ 4092734339 ┆ 3056 ┆ 50m 56s │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ ities/4092 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 734339.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 40.747234 ┆ -111.56201 ┆ … ┆ data/activ ┆ 5597638888 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 9 ┆ ┆ ities/5597 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 638888.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 40.916195 ┆ -111.86177 ┆ … ┆ data/activ ┆ 701803795 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 1 ┆ ┆ ities/7018 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 03795.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 46.914665 ┆ -121.64048 ┆ … ┆ data/activ ┆ 7475182621 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 8 ┆ ┆ ities/7475 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 182621.gpx ┆ ┆ ┆ │\n",
"│ null ┆ 0.0 ┆ 40.87928 ┆ -111.85519 ┆ … ┆ data/activ ┆ 5932628228 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ ┆ 4 ┆ ┆ ities/5932 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ 628228.gpx ┆ ┆ ┆ │\n",
"└────────┴────────────┴───────────┴────────────┴───┴────────────┴────────────┴─────────┴───────────┘"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# which rows are they?\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" .with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
" #.null_count()\n",
" .filter(pl.col('speed_between').is_null())\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Sgx009QZb0Qb",
"outputId": "c4d986bb-d374-4f9f-878d-9ff23a997014"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (1, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>elevation</th><th>speed_between</th><th>id</th><th>elapsed</th></tr><tr><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td><td>u32</td></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td><td>0</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (1, 7)\n",
"┌─────────────┬──────────┬───────────┬───────────┬───────────────┬─────┬─────────┐\n",
"│ distance_2d ┆ latitude ┆ longitude ┆ elevation ┆ speed_between ┆ id ┆ elapsed │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 ┆ u32 │\n",
"╞═════════════╪══════════╪═══════════╪═══════════╪═══════════════╪═════╪═════════╡\n",
"│ 0 ┆ 0 ┆ 0 ┆ 0 ┆ 0 ┆ 0 ┆ 0 │\n",
"└─────────────┴──────────┴───────────┴───────────┴───────────────┴─────┴─────────┘"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# what about nans?\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"\n",
"(raw\n",
" .select(cols)\n",
" # add elapsed time\n",
" .with_columns(id=calc_id)\n",
" # needs to be in its own with_columns because we reference id\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" #.null_count()\n",
" # no nan_count\n",
" #.filter(pl.col('speed_between').is_null())\n",
" .with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
" .select(cs.numeric().is_nan().sum())\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "FLWpICrbb0Qb"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "WzLbsRDhb0Qb"
},
"source": [
"## Create a function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 636
},
"id": "HuTDh0mFb0Qb",
"outputId": "d393b7bb-1d86-4b75-bb7f-f79b10bae494"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬─────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ str ┆ i64 ┆ i32 ┆ duration[ │\n",
"│ f32 ┆ ┆ ┆ μs] ┆ ┆ ┆ ┆ ┆ μs] │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:56 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 1 ┆ 1s │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:57 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 2 ┆ 2s │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:58 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 3 ┆ 3s │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:59 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 4 ┆ 4s │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:55:00 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4275 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 15s │\n",
"│ ┆ ┆ ┆ 16:01:42 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4276 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 16s │\n",
"│ ┆ ┆ ┆ 16:01:43 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4278 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 18s │\n",
"│ ┆ ┆ ┆ 16:01:45 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4281 ┆ 1h 11m │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 21s │\n",
"│ ┆ ┆ ┆ 16:01:48 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4282 ┆ 1h 11m │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 22s │\n",
"│ ┆ ┆ ┆ 16:01:49 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴─────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs]</td><td>f32</td><td>f32</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i32</td><td>duration[μs]</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;16:54:56</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;16:54:57</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;16:54:58</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;16:54:59</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;16:55:00</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:42</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:43</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:45</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;16:01:48</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;16:01:49</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 74
}
],
"source": [
"import polars as pl\n",
"import polars.selectors as cs\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"cols = ['course', 'distance_2d', 'latitude', 'longitude', 'time', 'elevation',\n",
" 'speed_between', 'name', 'type', 'filename']\n",
"# drop course because it is all null\n",
"final_cols = ['distance_2d', 'latitude', 'longitude', 'time', 'elevation', 'speed_between',\n",
" 'name', 'type', 'filename', 'id', 'elapsed', 'period']\n",
"\n",
"def tweak_strava(df):\n",
" return (df\n",
" .select(cols) # checking input\n",
" .with_columns(id=calc_id)\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" #pl.col(cat_cols).cast(pl.Categorical),\n",
" elapsed=calc_elapsed.cast(pl.Int32),\n",
" #time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" .with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
" .select(final_cols) # checking output\n",
" )\n",
"#file = '/content/drive/MyDrive/Colab Notebooks/clients/nvidia/GTC2024/data/activities.parquet'\n",
"file = 'activities_naive.parquet'\n",
"raw = pl.read_parquet(file)\n",
"tweak_strava(raw)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "VGAXTJ9fb0Qb"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "Pynwx0Ktb0Qb"
},
"source": [
"## Laziness & Chaining"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 683
},
"id": "pF9LMgz6b0Qb",
"outputId": "e965a99a-e09b-4a80-d812-04b000cc31a4"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 230 µs, sys: 52 µs, total: 282 µs\n",
"Wall time: 287 µs\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"<LazyFrame at 0x794FE57435D0>"
],
"text/html": [
"<h4>NAIVE QUERY PLAN</h4><p>run <b>LazyFrame.show_graph()</b> to see the optimized version</p><?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
"<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
" \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
"<!-- Generated by graphviz version 2.43.0 (0)\n",
" -->\n",
"<!-- Title: polars_query Pages: 1 -->\n",
"<svg width=\"1779pt\" height=\"406pt\"\n",
" viewBox=\"0.00 0.00 1779.00 406.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
"<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 402)\">\n",
"<title>polars_query</title>\n",
"<polygon fill=\"white\" stroke=\"transparent\" points=\"-4,4 -4,-402 1775,-402 1775,4 -4,4\"/>\n",
"<!-- p1 -->\n",
"<g id=\"node1\" class=\"node\">\n",
"<title>p1</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"914.5,-398 856.5,-398 856.5,-362 914.5,-362 914.5,-398\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-376.3\" font-family=\"Times,serif\" font-size=\"14.00\">π 12/12</text>\n",
"</g>\n",
"<!-- p2 -->\n",
"<g id=\"node2\" class=\"node\">\n",
"<title>p2</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1141.5,-326 629.5,-326 629.5,-290 1141.5,-290 1141.5,-326\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-304.3\" font-family=\"Times,serif\" font-size=\"14.00\">WITH COLUMNS [[(col(&quot;time&quot;)) &#45; (col(&quot;time&quot;).first())].over([col(&quot;id&quot;)]).alias(&quot;period&quot;)]</text>\n",
"</g>\n",
"<!-- p1&#45;&#45;p2 -->\n",
"<g id=\"edge1\" class=\"edge\">\n",
"<title>p1&#45;&#45;p2</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M885.5,-361.7C885.5,-350.85 885.5,-336.92 885.5,-326.1\"/>\n",
"</g>\n",
"<!-- p3 -->\n",
"<g id=\"node3\" class=\"node\">\n",
"<title>p3</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1771,-254 0,-254 0,-218 1771,-218 1771,-254\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-232.3\" font-family=\"Times,serif\" font-size=\"14.00\">WITH COLUMNS [col(&quot;distance_2d&quot;).strict_cast(Float32), col(&quot;latitude&quot;).strict_cast(Float32), col(&quot;longitude&quot;).strict_cast(Float32), col(&quot;elevation&quot;).strict_cast(Float32), col(&quot;speed_between&quot;).strict_cast(Float32), [(col(&quot;time&quot;)) &#45; (col(&quot;time&quot;).min())].dt.total_seconds().over([col(&quot;id&quot;)]).strict_cast(Int32).alias(&quot;elapsed&quot;)]</text>\n",
"</g>\n",
"<!-- p2&#45;&#45;p3 -->\n",
"<g id=\"edge2\" class=\"edge\">\n",
"<title>p2&#45;&#45;p3</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M885.5,-289.7C885.5,-278.85 885.5,-264.92 885.5,-254.1\"/>\n",
"</g>\n",
"<!-- p4 -->\n",
"<g id=\"node4\" class=\"node\">\n",
"<title>p4</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1282,-182 489,-182 489,-146 1282,-146 1282,-182\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-160.3\" font-family=\"Times,serif\" font-size=\"14.00\">WITH COLUMNS [col(&quot;filename&quot;).str.split([String(/)]).list.get([dyn int: &#45;1]).str.replace([String(.gpx), String()]).strict_cast(Int64).alias(&quot;id&quot;)]</text>\n",
"</g>\n",
"<!-- p3&#45;&#45;p4 -->\n",
"<g id=\"edge3\" class=\"edge\">\n",
"<title>p3&#45;&#45;p4</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M885.5,-217.7C885.5,-206.85 885.5,-192.92 885.5,-182.1\"/>\n",
"</g>\n",
"<!-- p5 -->\n",
"<g id=\"node5\" class=\"node\">\n",
"<title>p5</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"914.5,-110 856.5,-110 856.5,-74 914.5,-74 914.5,-110\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-88.3\" font-family=\"Times,serif\" font-size=\"14.00\">π 10/10</text>\n",
"</g>\n",
"<!-- p4&#45;&#45;p5 -->\n",
"<g id=\"edge4\" class=\"edge\">\n",
"<title>p4&#45;&#45;p5</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M885.5,-145.7C885.5,-134.85 885.5,-120.92 885.5,-110.1\"/>\n",
"</g>\n",
"<!-- p6 -->\n",
"<g id=\"node6\" class=\"node\">\n",
"<title>p6</title>\n",
"<polygon fill=\"none\" stroke=\"black\" points=\"1005,-38 766,-38 766,0 1005,0 1005,-38\"/>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-22.8\" font-family=\"Times,serif\" font-size=\"14.00\">Parquet SCAN [activities_naive.parquet]</text>\n",
"<text text-anchor=\"middle\" x=\"885.5\" y=\"-7.8\" font-family=\"Times,serif\" font-size=\"14.00\">π */10;</text>\n",
"</g>\n",
"<!-- p5&#45;&#45;p6 -->\n",
"<g id=\"edge5\" class=\"edge\">\n",
"<title>p5&#45;&#45;p6</title>\n",
"<path fill=\"none\" stroke=\"black\" d=\"M885.5,-73.81C885.5,-62.98 885.5,-49.01 885.5,-38.02\"/>\n",
"</g>\n",
"</g>\n",
"</svg>\n"
]
},
"metadata": {},
"execution_count": 75
}
],
"source": [
"%%time\n",
"raw_lazy = pl.scan_parquet(file)\n",
"tweak_strava(raw_lazy)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 671
},
"id": "KdPezDp8b0Qb",
"outputId": "fc7b6f79-d32d-4606-c81c-31290840b278"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 4.53 s, sys: 913 ms, total: 5.44 s\n",
"Wall time: 1.11 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬─────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ str ┆ i64 ┆ i32 ┆ duration[ │\n",
"│ f32 ┆ ┆ ┆ μs] ┆ ┆ ┆ ┆ ┆ μs] │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:56 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 1 ┆ 1s │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:57 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 2 ┆ 2s │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:58 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 3 ┆ 3s │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:59 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 4 ┆ 4s │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:55:00 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4275 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 15s │\n",
"│ ┆ ┆ ┆ 16:01:42 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4276 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 16s │\n",
"│ ┆ ┆ ┆ 16:01:43 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4278 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 18s │\n",
"│ ┆ ┆ ┆ 16:01:45 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4281 ┆ 1h 11m │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 21s │\n",
"│ ┆ ┆ ┆ 16:01:48 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4282 ┆ 1h 11m │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 22s │\n",
"│ ┆ ┆ ┆ 16:01:49 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴─────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs]</td><td>f32</td><td>f32</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i32</td><td>duration[μs]</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;16:54:56</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;16:54:57</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;16:54:58</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;16:54:59</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;16:55:00</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:42</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:43</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:45</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;16:01:48</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;16:01:49</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 76
}
],
"source": [
"%%time\n",
"tweak_strava(raw_lazy).collect()"
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"tweak_strava(raw_lazy).collect(engine='gpu')"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 671
},
"id": "C9CKVEjMdq5i",
"outputId": "b16aaad4-067e-4471-8be5-06bb637cf161"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 4.61 s, sys: 961 ms, total: 5.57 s\n",
"Wall time: 1.16 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬─────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ filename ┆ id ┆ elapsed ┆ period │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ str ┆ i64 ┆ i32 ┆ duration[ │\n",
"│ f32 ┆ ┆ ┆ μs] ┆ ┆ ┆ ┆ ┆ μs] │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 0 ┆ 0µs │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:56 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 1 ┆ 1s │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:57 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 2 ┆ 2s │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:58 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 3 ┆ 3s │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:54:59 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 4 ┆ 4s │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ │\n",
"│ ┆ ┆ ┆ 16:55:00 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4275 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 15s │\n",
"│ ┆ ┆ ┆ 16:01:42 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4276 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 16s │\n",
"│ ┆ ┆ ┆ 16:01:43 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4278 ┆ 1h 11m │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 18s │\n",
"│ ┆ ┆ ┆ 16:01:45 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4281 ┆ 1h 11m │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 21s │\n",
"│ ┆ ┆ ┆ 16:01:48 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4282 ┆ 1h 11m │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 22s │\n",
"│ ┆ ┆ ┆ 16:01:49 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ┆ ┆ gpx ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴─────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs]</td><td>f32</td><td>f32</td><td>str</td><td>str</td><td>str</td><td>i64</td><td>i32</td><td>duration[μs]</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;16:54:56</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;16:54:57</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;16:54:58</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;16:54:59</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;16:55:00</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:42</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:43</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;16:01:45</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;16:01:48</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;16:01:49</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 77
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 419
},
"id": "wYY1mG_Sb0Qb",
"outputId": "d9aac199-c717-4b37-850a-fd7db9683ddd"
},
"outputs": [
{
"output_type": "error",
"ename": "ComputeError",
"evalue": "'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Grouped rolling window not implemented\\n- NotImplementedError: Grouped rolling window not implemented\\n- NotImplementedError: Temporal function TemporalFunction.TotalSeconds\\n- NotImplementedError: list expr', [NotImplementedError('Grouped rolling window not implemented'), NotImplementedError('Grouped rolling window not implemented'), NotImplementedError('Temporal function TemporalFunction.TotalSeconds'), NotImplementedError('list expr')])",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mComputeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<timed eval>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.11/dist-packages/polars/lazyframe/frame.py\u001b[0m in \u001b[0;36mcollect\u001b[0;34m(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, streaming, engine, background, _eager, **_kwargs)\u001b[0m\n\u001b[1;32m 2027\u001b[0m \u001b[0;31m# Only for testing purposes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2028\u001b[0m \u001b[0mcallback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"post_opt_callback\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2029\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mwrap_df\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mldf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2030\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2031\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mComputeError\u001b[0m: 'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Grouped rolling window not implemented\\n- NotImplementedError: Grouped rolling window not implemented\\n- NotImplementedError: Temporal function TemporalFunction.TotalSeconds\\n- NotImplementedError: list expr', [NotImplementedError('Grouped rolling window not implemented'), NotImplementedError('Grouped rolling window not implemented'), NotImplementedError('Temporal function TemporalFunction.TotalSeconds'), NotImplementedError('list expr')])"
]
}
],
"source": [
"%%time\n",
"(tweak_strava(raw_lazy)\n",
" .collect(engine=pl.GPUEngine(device=0, raise_on_fail=True))\n",
")\n"
]
},
{
"cell_type": "code",
"source": [],
"metadata": {
"id": "FN5XGCKkfKq8"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 402
},
"outputId": "7f60475d-e04a-403f-c4b9-063e8d94ecef",
"id": "XI0oMCC1fLTX"
},
"outputs": [
{
"output_type": "error",
"ename": "ComputeError",
"evalue": "'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Time zone support', [NotImplementedError('Time zone support')])",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mComputeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<ipython-input-47-47852a1be1d0>\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 38\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 39\u001b[0m (tweak_strava3(raw_lazy)\n\u001b[0;32m---> 40\u001b[0;31m \u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mengine\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpl\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGPUEngine\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mraise_on_fail\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 41\u001b[0m )\n",
"\u001b[0;32m/usr/local/lib/python3.11/dist-packages/polars/lazyframe/frame.py\u001b[0m in \u001b[0;36mcollect\u001b[0;34m(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, streaming, engine, background, _eager, **_kwargs)\u001b[0m\n\u001b[1;32m 2027\u001b[0m \u001b[0;31m# Only for testing purposes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2028\u001b[0m \u001b[0mcallback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"post_opt_callback\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2029\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mwrap_df\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mldf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2030\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2031\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mComputeError\u001b[0m: 'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Time zone support', [NotImplementedError('Time zone support')])"
]
}
],
"source": [
"import polars as pl\n",
"import polars.selectors as cs\n",
"\n",
"calc_id = (pl.col('filename')\n",
" .str.split('/')\n",
" .list.get(-1)\n",
" .str.replace('.gpx', '')\n",
" .cast(pl.Int64)\n",
" )\n",
"calc_elapsed = (\n",
" (pl.col(\"time\") - pl.col(\"time\").min())\n",
" .dt.total_seconds()\n",
" .over(pl.col('id'))\n",
")\n",
"\n",
"float_cols = ['distance_2d', 'latitude', 'longitude', 'elevation', 'speed_between']\n",
"cat_cols = ['name', 'type', 'filename']\n",
"cols = ['course', 'distance_2d', 'latitude', 'longitude', 'time', 'elevation',\n",
" 'speed_between', 'name', 'type', 'filename']\n",
"# drop course because it is all null\n",
"final_cols = ['distance_2d', 'latitude', 'longitude', 'time', 'elevation', 'speed_between',\n",
" 'name', 'type', 'filename', #'id', #'elapsed', #'period'\n",
" ]\n",
"\n",
"def tweak_strava3(df):\n",
" return (df\n",
" .select(cols) # checking input\n",
" #.with_columns(id=calc_id)\n",
" .with_columns(\n",
" pl.col(float_cols).cast(pl.Float32),\n",
" #pl.col(cat_cols).cast(pl.Categorical),\n",
" #elapsed=calc_elapsed.cast(pl.Int32),\n",
" #time=pl.col('time').dt.convert_time_zone('America/Denver')\n",
" )\n",
" #.with_columns(period=(pl.col('time')-pl.col('time').first()).over(pl.col('id')))\n",
" .select(final_cols) # checking output\n",
" )\n",
"\n",
"(tweak_strava3(raw_lazy)\n",
" .collect(engine=pl.GPUEngine(device=0, raise_on_fail=True))\n",
")\n"
]
},
{
"cell_type": "code",
"source": [
"(tweak_strava(raw)\n",
"# make time naive - no tz\n",
" .with_columns(time=pl.col('time').dt.replace_time_zone(None))\n",
" .write_parquet('activities_naive.parquet')\n",
")\n"
],
"metadata": {
"id": "FjLumpqtgzsc"
},
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"source": [
"%%time\n",
"(tweak_strava3(raw_lazy)\n",
" .collect(engine=pl.GPUEngine(device=0, raise_on_fail=True))\n",
")\n"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 332
},
"id": "x_KpzlBYfS-D",
"outputId": "f4f17f24-1528-4802-88e2-a998ca6fdc20"
},
"execution_count": null,
"outputs": [
{
"output_type": "error",
"ename": "ComputeError",
"evalue": "'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Time zone support', [NotImplementedError('Time zone support')])",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mComputeError\u001b[0m Traceback (most recent call last)",
"\u001b[0;32m<timed eval>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
"\u001b[0;32m/usr/local/lib/python3.11/dist-packages/polars/lazyframe/frame.py\u001b[0m in \u001b[0;36mcollect\u001b[0;34m(self, type_coercion, predicate_pushdown, projection_pushdown, simplify_expression, slice_pushdown, comm_subplan_elim, comm_subexpr_elim, cluster_with_columns, collapse_joins, no_optimization, streaming, engine, background, _eager, **_kwargs)\u001b[0m\n\u001b[1;32m 2027\u001b[0m \u001b[0;31m# Only for testing purposes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2028\u001b[0m \u001b[0mcallback\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"post_opt_callback\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2029\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mwrap_df\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mldf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcollect\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 2030\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 2031\u001b[0m \u001b[0;34m@\u001b[0m\u001b[0moverload\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
"\u001b[0;31mComputeError\u001b[0m: 'cuda' conversion failed: NotImplementedError: ('Query execution with GPU not possible: unsupported operations.\\nThe errors were:\\n- NotImplementedError: Time zone support', [NotImplementedError('Time zone support')])"
]
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "dx9YR_sjb0Qb"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "S21b2NS5b0Qc"
},
"source": [
"## Stay in the Sandbox"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "-4z47KoJb0Qc",
"outputId": "059b6069-77f7-4622-eaca-fcc9ee645b00"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/var/folders/qn/r8_0pgj1645dn1w69vqls6cw0000gn/T/ipykernel_48473/370299276.py:6: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
" .with_columns(elevation_ft=pl.col('elevation').map_elements(meters_to_feet))\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n"
]
},
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 12)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>elevation_ft</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>f64</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29 10:54:56 MDT</td><td>1713.0</td><td>null</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>5620.07892</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29 10:54:57 MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>5619.750916</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29 10:54:58 MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>5619.422912</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29 10:54:59 MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>5619.422912</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29 10:55:00 MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365 and spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>5619.094508</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09 10:01:42 MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>4852.36236</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09 10:01:43 MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>4852.36236</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09 10:01:45 MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>4852.36236</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09 10:01:48 MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>4852.690364</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09 10:01:49 MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>4852.690364</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (5_050_370, 12)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬─────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ filename ┆ id ┆ elapsed ┆ elevation │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ _ft │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ cat ┆ i64 ┆ i32 ┆ --- │\n",
"│ f32 ┆ ┆ ┆ μs, Ameri ┆ ┆ ┆ ┆ ┆ f64 │\n",
"│ ┆ ┆ ┆ ca/Denver ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ] ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 0 ┆ 5620.0789 │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ 2 │\n",
"│ ┆ ┆ ┆ 10:54:56 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 1 ┆ 5619.7509 │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ 16 │\n",
"│ ┆ ┆ ┆ 10:54:57 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 2 ┆ 5619.4229 │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ 12 │\n",
"│ ┆ ┆ ┆ 10:54:58 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 3 ┆ 5619.4229 │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ 12 │\n",
"│ ┆ ┆ ┆ 10:54:59 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ data/acti ┆ 117698380 ┆ 4 ┆ 5619.0945 │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ vities/11 ┆ 26 ┆ ┆ 08 │\n",
"│ ┆ ┆ ┆ 10:55:00 ┆ ┆ 769838026 ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ .gp… ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4275 ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 6 │\n",
"│ ┆ ┆ ┆ 10:01:42 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4276 ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 6 │\n",
"│ ┆ ┆ ┆ 10:01:43 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4278 ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 6 │\n",
"│ ┆ ┆ ┆ 10:01:45 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4281 ┆ 4852.6903 │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 64 │\n",
"│ ┆ ┆ ┆ 10:01:48 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ gpx ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ data/acti ┆ 593262822 ┆ 4282 ┆ 4852.6903 │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ vities/59 ┆ 8 ┆ ┆ 64 │\n",
"│ ┆ ┆ ┆ 10:01:49 ┆ ┆ 32628228. ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ gpx ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴─────────┴───────────┘"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# convert elevation from meters to feet\n",
"def meters_to_feet(m):\n",
" return m * 3.28084\n",
"\n",
"(tweak_strava(raw)\n",
" .with_columns(elevation_ft=pl.col('elevation').map_elements(meters_to_feet))\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 654
},
"id": "xDi823Hfb0Qc",
"outputId": "f2c23607-c90a-47c2-bb53-28ef3f316b6d"
},
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 13)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬─────────┬───────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ id ┆ elapsed ┆ period ┆ elevation │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ _ft │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ i64 ┆ i32 ┆ duration[ ┆ --- │\n",
"│ f32 ┆ ┆ ┆ μs, Ameri ┆ ┆ ┆ ┆ μs] ┆ f32 │\n",
"│ ┆ ┆ ┆ ca/Denver ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ] ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═════════╪═══════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 0 ┆ 0µs ┆ 5620.0786 │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 13 │\n",
"│ ┆ ┆ ┆ 10:54:56 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 1 ┆ 1s ┆ 5619.7509 │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 77 │\n",
"│ ┆ ┆ ┆ 10:54:57 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 2 ┆ 2s ┆ 5619.4228 │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:58 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 3 ┆ 3s ┆ 5619.4228 │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:59 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 4 ┆ 4s ┆ 5619.0942 │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 38 │\n",
"│ ┆ ┆ ┆ 10:55:00 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4275 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 15s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:42 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4276 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 16s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:43 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4278 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 18s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:45 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4281 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ 8 ┆ ┆ 21s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:48 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4282 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ 8 ┆ ┆ 22s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:49 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴─────────┴───────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th><th>elevation_ft</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>duration[μs]</td><td>f32</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;10:54:56&nbsp;MDT</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td><td>5620.078613</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;10:54:57&nbsp;MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td><td>5619.750977</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;10:54:58&nbsp;MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td><td>5619.422852</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;10:54:59&nbsp;MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td><td>5619.422852</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;10:55:00&nbsp;MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td><td>5619.094238</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:42&nbsp;MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td><td>4852.362305</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:43&nbsp;MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td><td>4852.362305</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:45&nbsp;MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td><td>4852.362305</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;10:01:48&nbsp;MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td><td>4852.69043</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;10:01:49&nbsp;MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td><td>4852.69043</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 17
}
],
"source": [
"# convert elevation from meters to feet\n",
"def meters_to_feet(m):\n",
" return m * 3.28084\n",
"\n",
"(tweak_strava(raw)\n",
" .with_columns(elevation_ft=meters_to_feet(pl.col('elevation')))\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RfIKcwH8b0Qc",
"outputId": "c58508c7-52b2-4de1-e846-670921dab728"
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n",
"<magic-timeit>:2: PolarsInefficientMapWarning: \n",
"Expr.map_elements is significantly slower than the native expressions API.\n",
"Only use if you absolutely CANNOT implement your logic otherwise.\n",
"Replace this expression...\n",
" - pl.col(\"elevation\").map_elements(meters_to_feet)\n",
"with this one instead:\n",
" + pl.col(\"elevation\") * 3.28084\n",
"\n",
"sys:1: MapWithoutReturnDtypeWarning: Calling `map_elements` without specifying `return_dtype` can lead to unpredictable results. Specify `return_dtype` to silence this warning.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"745 ms ± 28.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
]
}
],
"source": [
"%%timeit\n",
"(tweak_strava(raw)\n",
" .with_columns(elevation_ft=pl.col('elevation').map_elements(meters_to_feet))\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Pw3cbcP2b0Qc",
"outputId": "7d20d480-2749-4bad-eea1-cc04aced69ac"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"448 ms ± 29.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
]
}
],
"source": [
"%%timeit\n",
"(tweak_strava(raw)\n",
" .with_columns(elevation_ft=meters_to_feet(pl.col('elevation')))\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Xfps7GwJb0Qc",
"outputId": "5ad48e36-7694-48fa-b9fe-ab8ec4e21941"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"442 ms ± 27.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
]
}
],
"source": [
"%%timeit\n",
"(tweak_strava(raw)\n",
" .with_columns(elevation_ft=pl.col('elevation') * 3.28084)\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Q6aUVm5Ab0Qc"
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 688
},
"id": "x3goNxelb0Qc",
"outputId": "6f149390-04ff-48dc-a036-e29b18d887af"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 3.66 s, sys: 1.16 s, total: 4.82 s\n",
"Wall time: 2.84 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 13)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬─────────┬───────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ id ┆ elapsed ┆ period ┆ elevation │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ _ft │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ i64 ┆ i32 ┆ duration[ ┆ --- │\n",
"│ f32 ┆ ┆ ┆ μs, Ameri ┆ ┆ ┆ ┆ μs] ┆ f32 │\n",
"│ ┆ ┆ ┆ ca/Denver ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ] ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═════════╪═══════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 0 ┆ 0µs ┆ 5620.0786 │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 13 │\n",
"│ ┆ ┆ ┆ 10:54:56 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 1 ┆ 1s ┆ 5619.7509 │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 77 │\n",
"│ ┆ ┆ ┆ 10:54:57 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 2 ┆ 2s ┆ 5619.4228 │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:58 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 3 ┆ 3s ┆ 5619.4228 │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:59 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 4 ┆ 4s ┆ 5619.0942 │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 38 │\n",
"│ ┆ ┆ ┆ 10:55:00 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4275 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 15s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:42 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4276 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 16s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:43 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4278 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 18s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:45 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4281 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ 8 ┆ ┆ 21s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:48 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4282 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ 8 ┆ ┆ 22s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:49 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴─────────┴───────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th><th>elevation_ft</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>duration[μs]</td><td>f32</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;10:54:56&nbsp;MDT</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td><td>5620.078613</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;10:54:57&nbsp;MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td><td>5619.750977</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;10:54:58&nbsp;MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td><td>5619.422852</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;10:54:59&nbsp;MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td><td>5619.422852</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;10:55:00&nbsp;MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td><td>5619.094238</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:42&nbsp;MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td><td>4852.362305</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:43&nbsp;MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td><td>4852.362305</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:45&nbsp;MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td><td>4852.362305</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;10:01:48&nbsp;MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td><td>4852.69043</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;10:01:49&nbsp;MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td><td>4852.69043</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 18
}
],
"source": [
"%%time\n",
"(tweak_strava(raw_lazy)\n",
" .with_columns(elevation_ft=meters_to_feet(pl.col('elevation')))\n",
" .collect()\n",
" )"
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"(tweak_strava(raw_lazy)\n",
" .with_columns(elevation_ft=meters_to_feet(pl.col('elevation')))\n",
" .collect(engine='gpu')\n",
" )"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 688
},
"id": "xgTppSHaeBEw",
"outputId": "05f3c138-d011-4045-a83e-c9ebd51b1955"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 3.71 s, sys: 1.08 s, total: 4.79 s\n",
"Wall time: 2.83 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (5_050_370, 13)\n",
"┌────────────┬───────────┬───────────┬───────────┬───┬───────────┬─────────┬───────────┬───────────┐\n",
"│ distance_2 ┆ latitude ┆ longitude ┆ time ┆ … ┆ id ┆ elapsed ┆ period ┆ elevation │\n",
"│ d ┆ --- ┆ --- ┆ --- ┆ ┆ --- ┆ --- ┆ --- ┆ _ft │\n",
"│ --- ┆ f32 ┆ f32 ┆ datetime[ ┆ ┆ i64 ┆ i32 ┆ duration[ ┆ --- │\n",
"│ f32 ┆ ┆ ┆ μs, Ameri ┆ ┆ ┆ ┆ μs] ┆ f32 │\n",
"│ ┆ ┆ ┆ ca/Denver ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ ] ┆ ┆ ┆ ┆ ┆ │\n",
"╞════════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═════════╪═══════════╪═══════════╡\n",
"│ 0.0 ┆ 41.370804 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 0 ┆ 0µs ┆ 5620.0786 │\n",
"│ ┆ ┆ 72 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 13 │\n",
"│ ┆ ┆ ┆ 10:54:56 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 6.704555 ┆ 41.370846 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 1 ┆ 1s ┆ 5619.7509 │\n",
"│ ┆ ┆ 11 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 77 │\n",
"│ ┆ ┆ ┆ 10:54:57 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4.160126 ┆ 41.370808 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 2 ┆ 2s ┆ 5619.4228 │\n",
"│ ┆ ┆ 18 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:58 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 2.62628 ┆ 41.370785 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 3 ┆ 3s ┆ 5619.4228 │\n",
"│ ┆ ┆ 26 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 52 │\n",
"│ ┆ ┆ ┆ 10:54:59 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.26929 ┆ 41.370773 ┆ -111.9017 ┆ 2024-06-2 ┆ … ┆ 117698380 ┆ 4 ┆ 4s ┆ 5619.0942 │\n",
"│ ┆ ┆ 33 ┆ 9 ┆ ┆ 26 ┆ ┆ ┆ 38 │\n",
"│ ┆ ┆ ┆ 10:55:00 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 0.344402 ┆ 40.87928 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4275 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 15s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:42 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 0.238017 ┆ 40.879284 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4276 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 16s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:43 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.05693 ┆ 40.879292 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4278 ┆ 1h 11m ┆ 4852.3623 │\n",
"│ ┆ ┆ 62 ┆ 9 ┆ ┆ 8 ┆ ┆ 18s ┆ 05 │\n",
"│ ┆ ┆ ┆ 10:01:45 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.259495 ┆ 40.879299 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4281 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 7 ┆ 9 ┆ ┆ 8 ┆ ┆ 21s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:48 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"│ 1.023777 ┆ 40.879311 ┆ -111.8553 ┆ 2021-09-0 ┆ … ┆ 593262822 ┆ 4282 ┆ 1h 11m ┆ 4852.6904 │\n",
"│ ┆ ┆ 77 ┆ 9 ┆ ┆ 8 ┆ ┆ 22s ┆ 3 │\n",
"│ ┆ ┆ ┆ 10:01:49 ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ ┆ ┆ MDT ┆ ┆ ┆ ┆ ┆ │\n",
"└────────────┴───────────┴───────────┴───────────┴───┴───────────┴─────────┴───────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (5_050_370, 13)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>distance_2d</th><th>latitude</th><th>longitude</th><th>time</th><th>elevation</th><th>speed_between</th><th>name</th><th>type</th><th>filename</th><th>id</th><th>elapsed</th><th>period</th><th>elevation_ft</th></tr><tr><td>f32</td><td>f32</td><td>f32</td><td>datetime[μs, America/Denver]</td><td>f32</td><td>f32</td><td>cat</td><td>cat</td><td>cat</td><td>i64</td><td>i32</td><td>duration[μs]</td><td>f32</td></tr></thead><tbody><tr><td>0.0</td><td>41.370804</td><td>-111.901772</td><td>2024-06-29&nbsp;10:54:56&nbsp;MDT</td><td>1713.0</td><td>null</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>0</td><td>0µs</td><td>5620.078613</td></tr><tr><td>6.704555</td><td>41.370846</td><td>-111.901711</td><td>2024-06-29&nbsp;10:54:57&nbsp;MDT</td><td>1712.900024</td><td>6.705301</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>1</td><td>1s</td><td>5619.750977</td></tr><tr><td>4.160126</td><td>41.370808</td><td>-111.901718</td><td>2024-06-29&nbsp;10:54:58&nbsp;MDT</td><td>1712.800049</td><td>4.161328</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>2</td><td>2s</td><td>5619.422852</td></tr><tr><td>2.62628</td><td>41.370785</td><td>-111.901726</td><td>2024-06-29&nbsp;10:54:59&nbsp;MDT</td><td>1712.800049</td><td>2.62628</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>3</td><td>3s</td><td>5619.422852</td></tr><tr><td>1.26929</td><td>41.370773</td><td>-111.901733</td><td>2024-06-29&nbsp;10:55:00&nbsp;MDT</td><td>1712.699951</td><td>1.273223</td><td>&quot;365&nbsp;and&nbsp;spork&quot;</td><td>&quot;ebikeride&quot;</td><td>&quot;data/activities/11769838026.gp…</td><td>11769838026</td><td>4</td><td>4s</td><td>5619.094238</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>0.344402</td><td>40.87928</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:42&nbsp;MDT</td><td>1479.0</td><td>0.172201</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4275</td><td>1h&nbsp;11m&nbsp;15s</td><td>4852.362305</td></tr><tr><td>0.238017</td><td>40.879284</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:43&nbsp;MDT</td><td>1479.0</td><td>0.238017</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4276</td><td>1h&nbsp;11m&nbsp;16s</td><td>4852.362305</td></tr><tr><td>1.05693</td><td>40.879292</td><td>-111.855362</td><td>2021-09-09&nbsp;10:01:45&nbsp;MDT</td><td>1479.0</td><td>0.528465</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4278</td><td>1h&nbsp;11m&nbsp;18s</td><td>4852.362305</td></tr><tr><td>1.259495</td><td>40.879299</td><td>-111.85537</td><td>2021-09-09&nbsp;10:01:48&nbsp;MDT</td><td>1479.099976</td><td>0.421153</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4281</td><td>1h&nbsp;11m&nbsp;21s</td><td>4852.69043</td></tr><tr><td>1.023777</td><td>40.879311</td><td>-111.855377</td><td>2021-09-09&nbsp;10:01:49&nbsp;MDT</td><td>1479.099976</td><td>1.023777</td><td>&quot;Morning&nbsp;walk&quot;</td><td>&quot;walking&quot;</td><td>&quot;data/activities/5932628228.gpx&quot;</td><td>5932628228</td><td>4282</td><td>1h&nbsp;11m&nbsp;22s</td><td>4852.69043</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 19
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "m3wb1yIQb0Qc"
},
"source": [
"## Master Aggregation"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "q8Lct9Ipb0Qc",
"outputId": "c5732576-e861-4ca2-e5a5-c3bba5f18e79"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (974, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>id</th><th>activity</th><th>distance</th><th>elevation</th><th>elapsed</th><th>period</th><th>speed</th></tr><tr><td>i64</td><td>cat</td><td>f32</td><td>f32</td><td>i32</td><td>duration[μs]</td><td>f64</td></tr></thead><tbody><tr><td>10861882731</td><td>&quot;Morning Run&quot;</td><td>41.345592</td><td>327619.0</td><td>134</td><td>2m 14s</td><td>0.308549</td></tr><tr><td>10074773869</td><td>&quot;Green canyon Ride (top loop is…</td><td>55.791634</td><td>7886.899902</td><td>3</td><td>3s</td><td>18.597211</td></tr><tr><td>4861795818</td><td>&quot;16 inches and forgot to record&quot;</td><td>110.950768</td><td>16022.299805</td><td>10</td><td>10s</td><td>11.095077</td></tr><tr><td>11265245273</td><td>&quot;Popperton&quot;</td><td>354.114624</td><td>316551.0625</td><td>211</td><td>3m 31s</td><td>1.678268</td></tr><tr><td>7475182669</td><td>&quot;X start of silver falls&quot;</td><td>494.293243</td><td>420216.46875</td><td>709</td><td>11m 49s</td><td>0.69717</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>3128639607</td><td>&quot;Quick solo&quot;</td><td>97783.789062</td><td>4856241.5</td><td>4625</td><td>1h 17m 5s</td><td>21.142441</td></tr><tr><td>10563801690</td><td>&quot;11 basin&quot;</td><td>102408.046875</td><td>2.68658752e8</td><td>165520</td><td>1d 21h 58m 40s</td><td>0.618705</td></tr><tr><td>3043338976</td><td>&quot;Ski with kids&quot;</td><td>164875.328125</td><td>2.387173e7</td><td>19000</td><td>5h 16m 40s</td><td>8.677649</td></tr><tr><td>12496788619</td><td>&quot;No stop&quot;</td><td>222882.515625</td><td>1.3668e6</td><td>82899</td><td>23h 1m 39s</td><td>2.688603</td></tr><tr><td>12379351421</td><td>&quot;Forgot to stop&quot;</td><td>449697.5625</td><td>6.0286624e7</td><td>197243</td><td>2d 6h 47m 23s</td><td>2.279916</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (974, 7)\n",
"┌─────────────┬───────────────┬───────────────┬──────────────┬─────────┬───────────────┬───────────┐\n",
"│ id ┆ activity ┆ distance ┆ elevation ┆ elapsed ┆ period ┆ speed │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ i64 ┆ cat ┆ f32 ┆ f32 ┆ i32 ┆ duration[μs] ┆ f64 │\n",
"╞═════════════╪═══════════════╪═══════════════╪══════════════╪═════════╪═══════════════╪═══════════╡\n",
"│ 10861882731 ┆ Morning Run ┆ 41.345592 ┆ 327619.0 ┆ 134 ┆ 2m 14s ┆ 0.308549 │\n",
"│ 10074773869 ┆ Green canyon ┆ 55.791634 ┆ 7886.899902 ┆ 3 ┆ 3s ┆ 18.597211 │\n",
"│ ┆ Ride (top ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ loop is… ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4861795818 ┆ 16 inches and ┆ 110.950768 ┆ 16022.299805 ┆ 10 ┆ 10s ┆ 11.095077 │\n",
"│ ┆ forgot to ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ record ┆ ┆ ┆ ┆ ┆ │\n",
"│ 11265245273 ┆ Popperton ┆ 354.114624 ┆ 316551.0625 ┆ 211 ┆ 3m 31s ┆ 1.678268 │\n",
"│ 7475182669 ┆ X start of ┆ 494.293243 ┆ 420216.46875 ┆ 709 ┆ 11m 49s ┆ 0.69717 │\n",
"│ ┆ silver falls ┆ ┆ ┆ ┆ ┆ │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 3128639607 ┆ Quick solo ┆ 97783.789062 ┆ 4856241.5 ┆ 4625 ┆ 1h 17m 5s ┆ 21.142441 │\n",
"│ 10563801690 ┆ 11 basin ┆ 102408.046875 ┆ 2.68658752e8 ┆ 165520 ┆ 1d 21h 58m ┆ 0.618705 │\n",
"│ ┆ ┆ ┆ ┆ ┆ 40s ┆ │\n",
"│ 3043338976 ┆ Ski with kids ┆ 164875.328125 ┆ 2.387173e7 ┆ 19000 ┆ 5h 16m 40s ┆ 8.677649 │\n",
"│ 12496788619 ┆ No stop ┆ 222882.515625 ┆ 1.3668e6 ┆ 82899 ┆ 23h 1m 39s ┆ 2.688603 │\n",
"│ 12379351421 ┆ Forgot to ┆ 449697.5625 ┆ 6.0286624e7 ┆ 197243 ┆ 2d 6h 47m 23s ┆ 2.279916 │\n",
"│ ┆ stop ┆ ┆ ┆ ┆ ┆ │\n",
"└─────────────┴───────────────┴───────────────┴──────────────┴─────────┴───────────────┴───────────┘"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# distance per activity\n",
"\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .sort('distance')\n",
" .collect()\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "8w72LwAdb0Qc",
"outputId": "d0587895-c2a9-446b-ea52-cdf586492abf"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (974, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>id</th><th>activity</th><th>distance</th><th>elevation</th><th>elapsed</th><th>period</th><th>speed</th></tr><tr><td>i64</td><td>cat</td><td>f32</td><td>f32</td><td>i32</td><td>duration[μs]</td><td>f64</td></tr></thead><tbody><tr><td>10074773869</td><td>&quot;Green canyon Ride (top loop is…</td><td>55.791634</td><td>7886.899902</td><td>3</td><td>3s</td><td>18.597211</td></tr><tr><td>4861795818</td><td>&quot;16 inches and forgot to record&quot;</td><td>110.950768</td><td>16022.299805</td><td>10</td><td>10s</td><td>11.095077</td></tr><tr><td>10861882731</td><td>&quot;Morning Run&quot;</td><td>41.345592</td><td>327619.0</td><td>134</td><td>2m 14s</td><td>0.308549</td></tr><tr><td>2260572607</td><td>&quot;Cruise ship level 4&quot;</td><td>1971.770386</td><td>0.0</td><td>191</td><td>3m 11s</td><td>10.323405</td></tr><tr><td>11265245273</td><td>&quot;Popperton&quot;</td><td>354.114624</td><td>316551.0625</td><td>211</td><td>3m 31s</td><td>1.678268</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>12496788619</td><td>&quot;No stop&quot;</td><td>222882.515625</td><td>1.3668e6</td><td>82899</td><td>23h 1m 39s</td><td>2.688603</td></tr><tr><td>9182951923</td><td>&quot;Two walks and one day&quot;</td><td>22229.818359</td><td>1.33492336e8</td><td>91989</td><td>1d 1h 33m 9s</td><td>0.241657</td></tr><tr><td>10563801690</td><td>&quot;11 basin&quot;</td><td>102408.046875</td><td>2.68658752e8</td><td>165520</td><td>1d 21h 58m 40s</td><td>0.618705</td></tr><tr><td>12379351421</td><td>&quot;Forgot to stop&quot;</td><td>449697.5625</td><td>6.0286624e7</td><td>197243</td><td>2d 6h 47m 23s</td><td>2.279916</td></tr><tr><td>3463601371</td><td>&quot;Grandeur peak&quot;</td><td>11213.65332</td><td>2.1661466e7</td><td>1238644328</td><td>14336d 3h 52m 8s</td><td>0.000009</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (974, 7)\n",
"┌─────────────┬──────────────┬──────────────┬──────────────┬────────────┬──────────────┬───────────┐\n",
"│ id ┆ activity ┆ distance ┆ elevation ┆ elapsed ┆ period ┆ speed │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ i64 ┆ cat ┆ f32 ┆ f32 ┆ i32 ┆ duration[μs] ┆ f64 │\n",
"╞═════════════╪══════════════╪══════════════╪══════════════╪════════════╪══════════════╪═══════════╡\n",
"│ 10074773869 ┆ Green canyon ┆ 55.791634 ┆ 7886.899902 ┆ 3 ┆ 3s ┆ 18.597211 │\n",
"│ ┆ Ride (top ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ loop is… ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4861795818 ┆ 16 inches ┆ 110.950768 ┆ 16022.299805 ┆ 10 ┆ 10s ┆ 11.095077 │\n",
"│ ┆ and forgot ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ to record ┆ ┆ ┆ ┆ ┆ │\n",
"│ 10861882731 ┆ Morning Run ┆ 41.345592 ┆ 327619.0 ┆ 134 ┆ 2m 14s ┆ 0.308549 │\n",
"│ 2260572607 ┆ Cruise ship ┆ 1971.770386 ┆ 0.0 ┆ 191 ┆ 3m 11s ┆ 10.323405 │\n",
"│ ┆ level 4 ┆ ┆ ┆ ┆ ┆ │\n",
"│ 11265245273 ┆ Popperton ┆ 354.114624 ┆ 316551.0625 ┆ 211 ┆ 3m 31s ┆ 1.678268 │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 12496788619 ┆ No stop ┆ 222882.51562 ┆ 1.3668e6 ┆ 82899 ┆ 23h 1m 39s ┆ 2.688603 │\n",
"│ ┆ ┆ 5 ┆ ┆ ┆ ┆ │\n",
"│ 9182951923 ┆ Two walks ┆ 22229.818359 ┆ 1.33492336e8 ┆ 91989 ┆ 1d 1h 33m 9s ┆ 0.241657 │\n",
"│ ┆ and one day ┆ ┆ ┆ ┆ ┆ │\n",
"│ 10563801690 ┆ 11 basin ┆ 102408.04687 ┆ 2.68658752e8 ┆ 165520 ┆ 1d 21h 58m ┆ 0.618705 │\n",
"│ ┆ ┆ 5 ┆ ┆ ┆ 40s ┆ │\n",
"│ 12379351421 ┆ Forgot to ┆ 449697.5625 ┆ 6.0286624e7 ┆ 197243 ┆ 2d 6h 47m ┆ 2.279916 │\n",
"│ ┆ stop ┆ ┆ ┆ ┆ 23s ┆ │\n",
"│ 3463601371 ┆ Grandeur ┆ 11213.65332 ┆ 2.1661466e7 ┆ 1238644328 ┆ 14336d 3h ┆ 0.000009 │\n",
"│ ┆ peak ┆ ┆ ┆ ┆ 52m 8s ┆ │\n",
"└─────────────┴──────────────┴──────────────┴──────────────┴────────────┴──────────────┴───────────┘"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# filter > 23 hours\n",
"\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .sort('elapsed')\n",
" .collect()\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 497
},
"id": "K6RK046wb0Qc",
"outputId": "fb3beebe-8f3e-4ffc-d76a-b8dc0e64265a"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 4.73 s, sys: 988 ms, total: 5.72 s\n",
"Wall time: 1.38 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (969, 7)\n",
"┌─────────────┬─────────────────┬──────────────┬──────────────┬─────────┬──────────────┬───────────┐\n",
"│ id ┆ activity ┆ distance ┆ elevation ┆ elapsed ┆ period ┆ speed │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ i64 ┆ str ┆ f32 ┆ f32 ┆ i32 ┆ duration[μs] ┆ f64 │\n",
"╞═════════════╪═════════════════╪══════════════╪══════════════╪═════════╪══════════════╪═══════════╡\n",
"│ 10074773869 ┆ Green canyon ┆ 55.791634 ┆ 7886.899902 ┆ 3 ┆ 3s ┆ 18.597211 │\n",
"│ ┆ Ride (top loop ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ is… ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4861795818 ┆ 16 inches and ┆ 110.950768 ┆ 16022.299805 ┆ 10 ┆ 10s ┆ 11.095077 │\n",
"│ ┆ forgot to ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ record ┆ ┆ ┆ ┆ ┆ │\n",
"│ 10861882731 ┆ Morning Run ┆ 41.345592 ┆ 327619.0 ┆ 134 ┆ 2m 14s ┆ 0.308549 │\n",
"│ 2260572607 ┆ Cruise ship ┆ 1971.770386 ┆ 0.0 ┆ 191 ┆ 3m 11s ┆ 10.323405 │\n",
"│ ┆ level 4 ┆ ┆ ┆ ┆ ┆ │\n",
"│ 11265245273 ┆ Popperton ┆ 354.114624 ┆ 316551.0625 ┆ 211 ┆ 3m 31s ┆ 1.678268 │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 723048762 ┆ Bear 100 ┆ 31935.894531 ┆ 1.6944826e7 ┆ 27494 ┆ 7h 38m 14s ┆ 1.161559 │\n",
"│ ┆ Evening Run ┆ ┆ ┆ ┆ ┆ │\n",
"│ 494446374 ┆ Park city with ┆ 52456.523438 ┆ 2.6608832e7 ┆ 27931 ┆ 7h 45m 31s ┆ 1.878075 │\n",
"│ ┆ kids ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13480080256 ┆ Brighton 15 ┆ 72800.445312 ┆ 8.4389008e7 ┆ 31277 ┆ 8h 41m 17s ┆ 2.327603 │\n",
"│ 8535457570 ┆ All day walk ┆ 17385.654297 ┆ 5.4682672e7 ┆ 40329 ┆ 11h 12m 9s ┆ 0.431096 │\n",
"│ 1020744327 ┆ Squaw peak ┆ 91396.789062 ┆ 4.3310584e7 ┆ 55955 ┆ 15h 32m 35s ┆ 1.633398 │\n",
"└─────────────┴─────────────────┴──────────────┴──────────────┴─────────┴──────────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (969, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>id</th><th>activity</th><th>distance</th><th>elevation</th><th>elapsed</th><th>period</th><th>speed</th></tr><tr><td>i64</td><td>str</td><td>f32</td><td>f32</td><td>i32</td><td>duration[μs]</td><td>f64</td></tr></thead><tbody><tr><td>10074773869</td><td>&quot;Green&nbsp;canyon&nbsp;Ride&nbsp;(top&nbsp;loop&nbsp;is…</td><td>55.791634</td><td>7886.899902</td><td>3</td><td>3s</td><td>18.597211</td></tr><tr><td>4861795818</td><td>&quot;16&nbsp;inches&nbsp;and&nbsp;forgot&nbsp;to&nbsp;record&quot;</td><td>110.950768</td><td>16022.299805</td><td>10</td><td>10s</td><td>11.095077</td></tr><tr><td>10861882731</td><td>&quot;Morning&nbsp;Run&quot;</td><td>41.345592</td><td>327619.0</td><td>134</td><td>2m&nbsp;14s</td><td>0.308549</td></tr><tr><td>2260572607</td><td>&quot;Cruise&nbsp;ship&nbsp;level&nbsp;4&quot;</td><td>1971.770386</td><td>0.0</td><td>191</td><td>3m&nbsp;11s</td><td>10.323405</td></tr><tr><td>11265245273</td><td>&quot;Popperton&quot;</td><td>354.114624</td><td>316551.0625</td><td>211</td><td>3m&nbsp;31s</td><td>1.678268</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>723048762</td><td>&quot;Bear&nbsp;100&nbsp;Evening&nbsp;Run&quot;</td><td>31935.894531</td><td>1.6944826e7</td><td>27494</td><td>7h&nbsp;38m&nbsp;14s</td><td>1.161559</td></tr><tr><td>494446374</td><td>&quot;Park&nbsp;city&nbsp;with&nbsp;kids&quot;</td><td>52456.523438</td><td>2.6608832e7</td><td>27931</td><td>7h&nbsp;45m&nbsp;31s</td><td>1.878075</td></tr><tr><td>13480080256</td><td>&quot;Brighton&nbsp;15&quot;</td><td>72800.445312</td><td>8.4389008e7</td><td>31277</td><td>8h&nbsp;41m&nbsp;17s</td><td>2.327603</td></tr><tr><td>8535457570</td><td>&quot;All&nbsp;day&nbsp;walk&quot;</td><td>17385.654297</td><td>5.4682672e7</td><td>40329</td><td>11h&nbsp;12m&nbsp;9s</td><td>0.431096</td></tr><tr><td>1020744327</td><td>&quot;Squaw&nbsp;peak&quot;</td><td>91396.789062</td><td>4.3310584e7</td><td>55955</td><td>15h&nbsp;32m&nbsp;35s</td><td>1.633398</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 79
}
],
"source": [
"%%time\n",
"# filter > 23 hours\n",
"\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .sort('elapsed')\n",
" .filter(pl.col('elapsed') < 23 * 60 * 60)\n",
" .collect()\n",
" )"
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"# filter > 23 hours\n",
"\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .sort('elapsed')\n",
" .filter(pl.col('elapsed') < 23 * 60 * 60)\n",
" .collect(engine='gpu')\n",
" )"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 497
},
"id": "Yroq7KF-eLo7",
"outputId": "b94388af-36f0-4466-aeaa-640d6bd3c02a"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 4.38 s, sys: 837 ms, total: 5.22 s\n",
"Wall time: 1.22 s\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (969, 7)\n",
"┌─────────────┬─────────────────┬──────────────┬──────────────┬─────────┬──────────────┬───────────┐\n",
"│ id ┆ activity ┆ distance ┆ elevation ┆ elapsed ┆ period ┆ speed │\n",
"│ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │\n",
"│ i64 ┆ str ┆ f32 ┆ f32 ┆ i32 ┆ duration[μs] ┆ f64 │\n",
"╞═════════════╪═════════════════╪══════════════╪══════════════╪═════════╪══════════════╪═══════════╡\n",
"│ 10074773869 ┆ Green canyon ┆ 55.791634 ┆ 7886.899902 ┆ 3 ┆ 3s ┆ 18.597211 │\n",
"│ ┆ Ride (top loop ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ is… ┆ ┆ ┆ ┆ ┆ │\n",
"│ 4861795818 ┆ 16 inches and ┆ 110.950768 ┆ 16022.299805 ┆ 10 ┆ 10s ┆ 11.095077 │\n",
"│ ┆ forgot to ┆ ┆ ┆ ┆ ┆ │\n",
"│ ┆ record ┆ ┆ ┆ ┆ ┆ │\n",
"│ 10861882731 ┆ Morning Run ┆ 41.345592 ┆ 327619.0 ┆ 134 ┆ 2m 14s ┆ 0.308549 │\n",
"│ 2260572607 ┆ Cruise ship ┆ 1971.770386 ┆ 0.0 ┆ 191 ┆ 3m 11s ┆ 10.323405 │\n",
"│ ┆ level 4 ┆ ┆ ┆ ┆ ┆ │\n",
"│ 11265245273 ┆ Popperton ┆ 354.114624 ┆ 316551.0625 ┆ 211 ┆ 3m 31s ┆ 1.678268 │\n",
"│ … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … │\n",
"│ 723048762 ┆ Bear 100 ┆ 31935.894531 ┆ 1.6944826e7 ┆ 27494 ┆ 7h 38m 14s ┆ 1.161559 │\n",
"│ ┆ Evening Run ┆ ┆ ┆ ┆ ┆ │\n",
"│ 494446374 ┆ Park city with ┆ 52456.523438 ┆ 2.6608832e7 ┆ 27931 ┆ 7h 45m 31s ┆ 1.878075 │\n",
"│ ┆ kids ┆ ┆ ┆ ┆ ┆ │\n",
"│ 13480080256 ┆ Brighton 15 ┆ 72800.445312 ┆ 8.4389008e7 ┆ 31277 ┆ 8h 41m 17s ┆ 2.327603 │\n",
"│ 8535457570 ┆ All day walk ┆ 17385.654297 ┆ 5.4682672e7 ┆ 40329 ┆ 11h 12m 9s ┆ 0.431096 │\n",
"│ 1020744327 ┆ Squaw peak ┆ 91396.789062 ┆ 4.3310584e7 ┆ 55955 ┆ 15h 32m 35s ┆ 1.633398 │\n",
"└─────────────┴─────────────────┴──────────────┴──────────────┴─────────┴──────────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (969, 7)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>id</th><th>activity</th><th>distance</th><th>elevation</th><th>elapsed</th><th>period</th><th>speed</th></tr><tr><td>i64</td><td>str</td><td>f32</td><td>f32</td><td>i32</td><td>duration[μs]</td><td>f64</td></tr></thead><tbody><tr><td>10074773869</td><td>&quot;Green&nbsp;canyon&nbsp;Ride&nbsp;(top&nbsp;loop&nbsp;is…</td><td>55.791634</td><td>7886.899902</td><td>3</td><td>3s</td><td>18.597211</td></tr><tr><td>4861795818</td><td>&quot;16&nbsp;inches&nbsp;and&nbsp;forgot&nbsp;to&nbsp;record&quot;</td><td>110.950768</td><td>16022.299805</td><td>10</td><td>10s</td><td>11.095077</td></tr><tr><td>10861882731</td><td>&quot;Morning&nbsp;Run&quot;</td><td>41.345592</td><td>327619.0</td><td>134</td><td>2m&nbsp;14s</td><td>0.308549</td></tr><tr><td>2260572607</td><td>&quot;Cruise&nbsp;ship&nbsp;level&nbsp;4&quot;</td><td>1971.770386</td><td>0.0</td><td>191</td><td>3m&nbsp;11s</td><td>10.323405</td></tr><tr><td>11265245273</td><td>&quot;Popperton&quot;</td><td>354.114624</td><td>316551.0625</td><td>211</td><td>3m&nbsp;31s</td><td>1.678268</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>723048762</td><td>&quot;Bear&nbsp;100&nbsp;Evening&nbsp;Run&quot;</td><td>31935.894531</td><td>1.6944826e7</td><td>27494</td><td>7h&nbsp;38m&nbsp;14s</td><td>1.161559</td></tr><tr><td>494446374</td><td>&quot;Park&nbsp;city&nbsp;with&nbsp;kids&quot;</td><td>52456.523438</td><td>2.6608832e7</td><td>27931</td><td>7h&nbsp;45m&nbsp;31s</td><td>1.878075</td></tr><tr><td>13480080256</td><td>&quot;Brighton&nbsp;15&quot;</td><td>72800.445312</td><td>8.4389008e7</td><td>31277</td><td>8h&nbsp;41m&nbsp;17s</td><td>2.327603</td></tr><tr><td>8535457570</td><td>&quot;All&nbsp;day&nbsp;walk&quot;</td><td>17385.654297</td><td>5.4682672e7</td><td>40329</td><td>11h&nbsp;12m&nbsp;9s</td><td>0.431096</td></tr><tr><td>1020744327</td><td>&quot;Squaw&nbsp;peak&quot;</td><td>91396.789062</td><td>4.3310584e7</td><td>55955</td><td>15h&nbsp;32m&nbsp;35s</td><td>1.633398</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 80
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "fmwcGYfib0Qc"
},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "c2OC6vHVb0Qc",
"outputId": "60b677c0-22cc-4ebd-aca3-6c55c26468aa"
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"<style>\n",
" #altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a.vega-embed {\n",
" width: 100%;\n",
" display: flex;\n",
" }\n",
"\n",
" #altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a.vega-embed details,\n",
" #altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a.vega-embed details summary {\n",
" position: relative;\n",
" }\n",
"</style>\n",
"<div id=\"altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a\"></div>\n",
"<script type=\"text/javascript\">\n",
" var VEGA_DEBUG = (typeof VEGA_DEBUG == \"undefined\") ? {} : VEGA_DEBUG;\n",
" (function(spec, embedOpt){\n",
" let outputDiv = document.currentScript.previousElementSibling;\n",
" if (outputDiv.id !== \"altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a\") {\n",
" outputDiv = document.getElementById(\"altair-viz-5c4ee0ea006340f4acb0c56d0ea7c06a\");\n",
" }\n",
" const paths = {\n",
" \"vega\": \"https://cdn.jsdelivr.net/npm/vega@5?noext\",\n",
" \"vega-lib\": \"https://cdn.jsdelivr.net/npm/vega-lib?noext\",\n",
" \"vega-lite\": \"https://cdn.jsdelivr.net/npm/[email protected]?noext\",\n",
" \"vega-embed\": \"https://cdn.jsdelivr.net/npm/vega-embed@6?noext\",\n",
" };\n",
"\n",
" function maybeLoadScript(lib, version) {\n",
" var key = `${lib.replace(\"-\", \"\")}_version`;\n",
" return (VEGA_DEBUG[key] == version) ?\n",
" Promise.resolve(paths[lib]) :\n",
" new Promise(function(resolve, reject) {\n",
" var s = document.createElement('script');\n",
" document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
" s.async = true;\n",
" s.onload = () => {\n",
" VEGA_DEBUG[key] = version;\n",
" return resolve(paths[lib]);\n",
" };\n",
" s.onerror = () => reject(`Error loading script: ${paths[lib]}`);\n",
" s.src = paths[lib];\n",
" });\n",
" }\n",
"\n",
" function showError(err) {\n",
" outputDiv.innerHTML = `<div class=\"error\" style=\"color:red;\">${err}</div>`;\n",
" throw err;\n",
" }\n",
"\n",
" function displayChart(vegaEmbed) {\n",
" vegaEmbed(outputDiv, spec, embedOpt)\n",
" .catch(err => showError(`Javascript Error: ${err.message}<br>This usually means there's a typo in your chart specification. See the javascript console for the full traceback.`));\n",
" }\n",
"\n",
" if(typeof define === \"function\" && define.amd) {\n",
" requirejs.config({paths});\n",
" require([\"vega-embed\"], displayChart, err => showError(`Error loading script: ${err.message}`));\n",
" } else {\n",
" maybeLoadScript(\"vega\", \"5\")\n",
" .then(() => maybeLoadScript(\"vega-lite\", \"5.20.1\"))\n",
" .then(() => maybeLoadScript(\"vega-embed\", \"6\"))\n",
" .catch(showError)\n",
" .then(() => displayChart(vegaEmbed));\n",
" }\n",
" })({\"config\": {\"view\": {\"continuousWidth\": 300, \"continuousHeight\": 300}}, \"data\": {\"name\": \"data-58e608f08a6c1059b46d2eb45b101e6b\"}, \"mark\": {\"type\": \"bar\", \"tooltip\": true}, \"encoding\": {\"x\": {\"field\": \"distance\", \"type\": \"quantitative\"}, \"y\": {\"field\": \"activity\", \"type\": \"nominal\"}}, \"params\": [{\"name\": \"param_8\", \"select\": {\"type\": \"interval\", \"encodings\": [\"x\", \"y\"]}, \"bind\": \"scales\"}], \"$schema\": \"https://vega.github.io/schema/vega-lite/v5.20.1.json\", \"datasets\": {\"data-58e608f08a6c1059b46d2eb45b101e6b\": [{\"activity\": \"Solitude 22\", \"distance\": 54896.69140625}, {\"activity\": \"Solo powder\", \"distance\": 55164.13671875}, {\"activity\": \"4 Basin\", \"distance\": 56134.09765625}, {\"activity\": \"Deer valley 2\", \"distance\": 57012.9453125}, {\"activity\": \"Day 2\", \"distance\": 57643.4140625}, {\"activity\": \"Ski with kids\", \"distance\": 59545.52734375}, {\"activity\": \"20 basin\", \"distance\": 59694.13671875}, {\"activity\": \"Birthday with Olivia\", \"distance\": 59722.36328125}, {\"activity\": \"Snowbasin\", \"distance\": 60681.828125}, {\"activity\": \"Olivia birthday ski Park City\", \"distance\": 63158.02734375}, {\"activity\": \"Snowbasin James and Norah\", \"distance\": 67403.859375}, {\"activity\": \"Msnowbasin\", \"distance\": 68454.0078125}, {\"activity\": \"Brighton 15\", \"distance\": 72800.4453125}, {\"activity\": \"Family ski day\", \"distance\": 75762.828125}, {\"activity\": \"Family day\", \"distance\": 77317.8359375}, {\"activity\": \"28 Brighton\", \"distance\": 78870.984375}, {\"activity\": \"Alta with Andrea\", \"distance\": 84568.515625}, {\"activity\": \"Squaw peak\", \"distance\": 91396.7890625}, {\"activity\": \"Quick solo\", \"distance\": 97783.7890625}, {\"activity\": \"Ski with kids\", \"distance\": 164875.328125}]}}, {\"mode\": \"vega-lite\"});\n",
"</script>"
],
"text/plain": [
"alt.Chart(...)"
]
},
"execution_count": 41,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# plot\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .filter(pl.col('elapsed') < 23 * 60 * 60)\n",
" .sort('distance')\n",
"\n",
" .tail(20)\n",
" .collect()\n",
" .select([pl.col('activity').cast(pl.Utf8), 'distance', ])\n",
" .plot.bar(y='activity', x='distance')\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "mQApUHK8b0Qc",
"outputId": "c6ae21c0-eb4e-4ac9-d417-a52d036c8fb0"
},
"outputs": [
{
"data": {},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.holoviews_exec.v0+json": "",
"text/html": [
"<div id='p1014'>\n",
" <div id=\"b3a40334-8c3f-4a42-aa7f-ae1426964088\" data-root-id=\"p1014\" style=\"display: contents;\"></div>\n",
"</div>\n",
"<script type=\"application/javascript\">(function(root) {\n",
" var docs_json = {\"6897a98a-f233-4798-b2b9-9d68ea15f0d5\":{\"version\":\"3.3.4\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"Row\",\"id\":\"p1014\",\"attributes\":{\"name\":\"Row01179\",\"tags\":[\"embedded\"],\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1017\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/loading.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1073\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/listpanel.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1015\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/default.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1016\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/native.css\"}}],\"min_width\":700,\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\",\"children\":[{\"type\":\"object\",\"name\":\"Spacer\",\"id\":\"p1018\",\"attributes\":{\"name\":\"HSpacer01186\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"p1017\"},{\"id\":\"p1015\"},{\"id\":\"p1016\"}],\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\"}},{\"type\":\"object\",\"name\":\"Figure\",\"id\":\"p1026\",\"attributes\":{\"width\":700,\"height\":300,\"margin\":[5,10],\"sizing_mode\":\"fixed\",\"align\":\"start\",\"x_range\":{\"type\":\"object\",\"name\":\"Range1d\",\"id\":\"p1019\",\"attributes\":{\"tags\":[[[\"distance\",\"distance\",null]],[]],\"end\":175873.191796875,\"reset_start\":0.0,\"reset_end\":175873.191796875}},\"y_range\":{\"type\":\"object\",\"name\":\"FactorRange\",\"id\":\"p1020\",\"attributes\":{\"tags\":[[[\"activity\",\"activity\",null]],{\"type\":\"map\",\"entries\":[[\"invert_yaxis\",false],[\"autorange\",false]]}],\"factors\":[\"Solitude 22\",\"Solo powder\",\"4 Basin\",\"Deer valley 2\",\"Day 2\",\"Ski with kids\",\"20 basin\",\"Birthday with Olivia\",\"Snowbasin\",\"Olivia birthday ski Park City\",\"Snowbasin James and Norah\",\"Msnowbasin\",\"Brighton 15\",\"Family ski day\",\"Family day\",\"28 Brighton\",\"Alta with Andrea\",\"Squaw peak\",\"Quick solo\"]}},\"x_scale\":{\"type\":\"object\",\"name\":\"LinearScale\",\"id\":\"p1036\"},\"y_scale\":{\"type\":\"object\",\"name\":\"CategoricalScale\",\"id\":\"p1037\"},\"title\":{\"type\":\"object\",\"name\":\"Title\",\"id\":\"p1029\",\"attributes\":{\"text_color\":\"black\",\"text_font_size\":\"12pt\"}},\"renderers\":[{\"type\":\"object\",\"name\":\"GlyphRenderer\",\"id\":\"p1066\",\"attributes\":{\"data_source\":{\"type\":\"object\",\"name\":\"ColumnDataSource\",\"id\":\"p1057\",\"attributes\":{\"selected\":{\"type\":\"object\",\"name\":\"Selection\",\"id\":\"p1058\",\"attributes\":{\"indices\":[],\"line_indices\":[]}},\"selection_policy\":{\"type\":\"object\",\"name\":\"UnionRenderers\",\"id\":\"p1059\"},\"data\":{\"type\":\"map\",\"entries\":[[\"activity\",[\"Solitude 22\",\"Solo powder\",\"4 Basin\",\"Deer valley 2\",\"Day 2\",\"Ski with kids\",\"20 basin\",\"Birthday with Olivia\",\"Snowbasin\",\"Olivia birthday ski Park City\",\"Snowbasin James and Norah\",\"Msnowbasin\",\"Brighton 15\",\"Family ski day\",\"Family day\",\"28 Brighton\",\"Alta with Andrea\",\"Squaw peak\",\"Quick solo\",\"Ski with kids\"]],[\"distance\",{\"type\":\"ndarray\",\"array\":{\"type\":\"bytes\",\"data\":\"sXBWRyN8V0cZRltH8rReR2orYUeHmWhHIy5pR11KaUfUCW1HB7Z2R+6lg0cBs4VHOTCOR2r5k0frApdHfguaR0IspUdlgrJH5fu+R9UCIUg=\"},\"shape\":[20],\"dtype\":\"float32\",\"order\":\"little\"}]]}}},\"view\":{\"type\":\"object\",\"name\":\"CDSView\",\"id\":\"p1067\",\"attributes\":{\"filter\":{\"type\":\"object\",\"name\":\"AllIndices\",\"id\":\"p1068\"}}},\"glyph\":{\"type\":\"object\",\"name\":\"HBar\",\"id\":\"p1063\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"y\":{\"type\":\"field\",\"field\":\"activity\"},\"height\":{\"type\":\"value\",\"value\":0.8},\"right\":{\"type\":\"field\",\"field\":\"distance\"},\"fill_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"hatch_color\":{\"type\":\"value\",\"value\":\"#30a2da\"}}},\"selection_glyph\":{\"type\":\"object\",\"name\":\"HBar\",\"id\":\"p1069\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"y\":{\"type\":\"field\",\"field\":\"activity\"},\"height\":{\"type\":\"value\",\"value\":0.8},\"left\":{\"type\":\"value\",\"value\":0},\"right\":{\"type\":\"field\",\"field\":\"distance\"},\"line_color\":{\"type\":\"value\",\"value\":\"black\"},\"line_alpha\":{\"type\":\"value\",\"value\":1.0},\"line_width\":{\"type\":\"value\",\"value\":1},\"line_join\":{\"type\":\"value\",\"value\":\"bevel\"},\"line_cap\":{\"type\":\"value\",\"value\":\"butt\"},\"line_dash\":{\"type\":\"value\",\"value\":[]},\"line_dash_offset\":{\"type\":\"value\",\"value\":0},\"fill_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"fill_alpha\":{\"type\":\"value\",\"value\":1.0},\"hatch_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"hatch_alpha\":{\"type\":\"value\",\"value\":1.0},\"hatch_scale\":{\"type\":\"value\",\"value\":12.0},\"hatch_pattern\":{\"type\":\"value\",\"value\":null},\"hatch_weight\":{\"type\":\"value\",\"value\":1.0}}},\"nonselection_glyph\":{\"type\":\"object\",\"name\":\"HBar\",\"id\":\"p1064\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"y\":{\"type\":\"field\",\"field\":\"activity\"},\"height\":{\"type\":\"value\",\"value\":0.8},\"right\":{\"type\":\"field\",\"field\":\"distance\"},\"line_alpha\":{\"type\":\"value\",\"value\":0.1},\"fill_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"fill_alpha\":{\"type\":\"value\",\"value\":0.1},\"hatch_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"hatch_alpha\":{\"type\":\"value\",\"value\":0.1}}},\"muted_glyph\":{\"type\":\"object\",\"name\":\"HBar\",\"id\":\"p1065\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"y\":{\"type\":\"field\",\"field\":\"activity\"},\"height\":{\"type\":\"value\",\"value\":0.8},\"right\":{\"type\":\"field\",\"field\":\"distance\"},\"line_alpha\":{\"type\":\"value\",\"value\":0.2},\"fill_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"fill_alpha\":{\"type\":\"value\",\"value\":0.2},\"hatch_color\":{\"type\":\"value\",\"value\":\"#30a2da\"},\"hatch_alpha\":{\"type\":\"value\",\"value\":0.2}}}}}],\"toolbar\":{\"type\":\"object\",\"name\":\"Toolbar\",\"id\":\"p1035\",\"attributes\":{\"tools\":[{\"type\":\"object\",\"name\":\"WheelZoomTool\",\"id\":\"p1024\",\"attributes\":{\"tags\":[\"hv_created\"],\"renderers\":\"auto\",\"zoom_together\":\"none\"}},{\"type\":\"object\",\"name\":\"HoverTool\",\"id\":\"p1025\",\"attributes\":{\"tags\":[\"hv_created\"],\"renderers\":[{\"id\":\"p1066\"}],\"tooltips\":[[\"activity\",\"@{activity}\"],[\"distance\",\"@{distance}\"]]}},{\"type\":\"object\",\"name\":\"SaveTool\",\"id\":\"p1048\"},{\"type\":\"object\",\"name\":\"PanTool\",\"id\":\"p1049\"},{\"type\":\"object\",\"name\":\"BoxZoomTool\",\"id\":\"p1050\",\"attributes\":{\"overlay\":{\"type\":\"object\",\"name\":\"BoxAnnotation\",\"id\":\"p1051\",\"attributes\":{\"syncable\":false,\"level\":\"overlay\",\"visible\":false,\"left\":{\"type\":\"number\",\"value\":\"nan\"},\"right\":{\"type\":\"number\",\"value\":\"nan\"},\"top\":{\"type\":\"number\",\"value\":\"nan\"},\"bottom\":{\"type\":\"number\",\"value\":\"nan\"},\"left_units\":\"canvas\",\"right_units\":\"canvas\",\"top_units\":\"canvas\",\"bottom_units\":\"canvas\",\"line_color\":\"black\",\"line_alpha\":1.0,\"line_width\":2,\"line_dash\":[4,4],\"fill_color\":\"lightgrey\",\"fill_alpha\":0.5}}}},{\"type\":\"object\",\"name\":\"ResetTool\",\"id\":\"p1056\"}],\"active_drag\":{\"id\":\"p1049\"},\"active_scroll\":{\"id\":\"p1024\"}}},\"left\":[{\"type\":\"object\",\"name\":\"CategoricalAxis\",\"id\":\"p1043\",\"attributes\":{\"ticker\":{\"type\":\"object\",\"name\":\"CategoricalTicker\",\"id\":\"p1044\"},\"formatter\":{\"type\":\"object\",\"name\":\"CategoricalTickFormatter\",\"id\":\"p1045\"},\"axis_label\":\"activity\",\"major_label_policy\":{\"type\":\"object\",\"name\":\"AllLabels\",\"id\":\"p1046\"}}}],\"below\":[{\"type\":\"object\",\"name\":\"LinearAxis\",\"id\":\"p1038\",\"attributes\":{\"ticker\":{\"type\":\"object\",\"name\":\"BasicTicker\",\"id\":\"p1039\",\"attributes\":{\"mantissas\":[1,2,5]}},\"formatter\":{\"type\":\"object\",\"name\":\"BasicTickFormatter\",\"id\":\"p1040\"},\"axis_label\":\"distance\",\"major_label_policy\":{\"type\":\"object\",\"name\":\"AllLabels\",\"id\":\"p1041\"}}}],\"center\":[{\"type\":\"object\",\"name\":\"Grid\",\"id\":\"p1042\",\"attributes\":{\"axis\":{\"id\":\"p1038\"},\"grid_line_color\":null}},{\"type\":\"object\",\"name\":\"Grid\",\"id\":\"p1047\",\"attributes\":{\"dimension\":1,\"axis\":{\"id\":\"p1043\"},\"grid_line_color\":null}}],\"min_border_top\":10,\"min_border_bottom\":10,\"min_border_left\":10,\"min_border_right\":10,\"output_backend\":\"webgl\"}},{\"type\":\"object\",\"name\":\"Spacer\",\"id\":\"p1071\",\"attributes\":{\"name\":\"HSpacer01187\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"p1017\"},{\"id\":\"p1015\"},{\"id\":\"p1016\"}],\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\"}}]}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"copy_to_clipboard1\",\"properties\":[{\"name\":\"fill\",\"kind\":\"Any\",\"default\":\"none\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
" var render_items = [{\"docid\":\"6897a98a-f233-4798-b2b9-9d68ea15f0d5\",\"roots\":{\"p1014\":\"b3a40334-8c3f-4a42-aa7f-ae1426964088\"},\"root_ids\":[\"p1014\"]}];\n",
" var docs = Object.values(docs_json)\n",
" if (!docs) {\n",
" return\n",
" }\n",
" const py_version = docs[0].version.replace('rc', '-rc.').replace('.dev', '-dev.')\n",
" function embed_document(root) {\n",
" var Bokeh = get_bokeh(root)\n",
" Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
" for (const render_item of render_items) {\n",
" for (const root_id of render_item.root_ids) {\n",
"\tconst id_el = document.getElementById(root_id)\n",
"\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
"\t const root_el = id_el.children[0]\n",
"\t root_el.id = root_el.id + '-rendered'\n",
"\t}\n",
" }\n",
" }\n",
" }\n",
" function get_bokeh(root) {\n",
" if (root.Bokeh === undefined) {\n",
" return null\n",
" } else if (root.Bokeh.version !== py_version) {\n",
" if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
"\treturn null\n",
" }\n",
" return root.Bokeh.versions.get(py_version);\n",
" } else if (root.Bokeh.version === py_version) {\n",
" return root.Bokeh\n",
" }\n",
" return null\n",
" }\n",
" function is_loaded(root) {\n",
" var Bokeh = get_bokeh(root)\n",
" return (Bokeh != null && Bokeh.Panel !== undefined)\n",
" }\n",
" if (is_loaded(root)) {\n",
" embed_document(root);\n",
" } else {\n",
" var attempts = 0;\n",
" var timer = setInterval(function(root) {\n",
" if (is_loaded(root)) {\n",
" clearInterval(timer);\n",
" embed_document(root);\n",
" } else if (document.readyState == \"complete\") {\n",
" attempts++;\n",
" if (attempts > 200) {\n",
" clearInterval(timer);\n",
"\t var Bokeh = get_bokeh(root)\n",
"\t if (Bokeh == null || Bokeh.Panel == null) {\n",
" console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
"\t } else {\n",
"\t console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
"\t embed_document(root)\n",
"\t }\n",
" }\n",
" }\n",
" }, 25, root)\n",
" }\n",
"})(window);</script>"
],
"text/plain": [
":Bars [activity] (distance)"
]
},
"execution_count": 71,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {
"id": "p1014"
}
},
"output_type": "execute_result"
}
],
"source": [
"import hvplot.polars\n",
"# plot\n",
"(tweak_strava(raw_lazy)\n",
" .group_by('id')\n",
" .agg(\n",
" activity=pl.col('name').first(),\n",
" distance=pl.col('distance_2d').sum(),\n",
" elevation=pl.col('elevation').sum(),\n",
" elapsed=pl.col('elapsed').last(),\n",
" period=pl.col('period').last(),\n",
" )\n",
" .with_columns(speed=pl.col('distance') / pl.col('elapsed'))\n",
" .filter(pl.col('elapsed') < 23 * 60 * 60)\n",
" .sort('distance')\n",
"\n",
" .tail(20)\n",
" .collect()\n",
" .select([pl.col('activity').cast(pl.Utf8), 'distance', ])\n",
" .hvplot.barh(x='activity', y='distance')\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "jXRuXwh-b0Qc",
"outputId": "57854a8d-e70c-44e8-e9e9-4aa00324f43d"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (11, 1)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>type</th></tr><tr><td>struct[2]</td></tr></thead><tbody><tr><td>{&quot;ebikeride&quot;,433738}</td></tr><tr><td>{&quot;cycling&quot;,784828}</td></tr><tr><td>{&quot;Snowshoe&quot;,9938}</td></tr><tr><td>{&quot;AlpineSki&quot;,2227436}</td></tr><tr><td>{&quot;running&quot;,738214}</td></tr><tr><td>{&quot;hiking&quot;,255803}</td></tr><tr><td>{&quot;walking&quot;,524654}</td></tr><tr><td>{&quot;BackcountrySki&quot;,35337}</td></tr><tr><td>{&quot;StandUpPaddling&quot;,33020}</td></tr><tr><td>{&quot;Kayaking&quot;,3580}</td></tr><tr><td>{&quot;NordicSki&quot;,3822}</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (11, 1)\n",
"┌───────────────────────────┐\n",
"│ type │\n",
"│ --- │\n",
"│ struct[2] │\n",
"╞═══════════════════════════╡\n",
"│ {\"ebikeride\",433738} │\n",
"│ {\"cycling\",784828} │\n",
"│ {\"Snowshoe\",9938} │\n",
"│ {\"AlpineSki\",2227436} │\n",
"│ {\"running\",738214} │\n",
"│ {\"hiking\",255803} │\n",
"│ {\"walking\",524654} │\n",
"│ {\"BackcountrySki\",35337} │\n",
"│ {\"StandUpPaddling\",33020} │\n",
"│ {\"Kayaking\",3580} │\n",
"│ {\"NordicSki\",3822} │\n",
"└───────────────────────────┘"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"with pl.Config(set_tbl_rows=20):\n",
" display(tweak_strava(raw_lazy)\n",
" .select(pl.col('type').value_counts())\n",
" .collect()\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "wDGS9dStb0Qc",
"outputId": "72f0a391-3a24-4d41-c501-a6595858abdf"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (11, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>year</th><th>type</th><th>distance</th></tr><tr><td>i32</td><td>cat</td><td>f32</td></tr></thead><tbody><tr><td>2022</td><td>&quot;cycling&quot;</td><td>479097.40625</td></tr><tr><td>2018</td><td>&quot;cycling&quot;</td><td>5883.088379</td></tr><tr><td>2016</td><td>&quot;cycling&quot;</td><td>26981.732422</td></tr><tr><td>2021</td><td>&quot;cycling&quot;</td><td>413450.9375</td></tr><tr><td>2019</td><td>&quot;cycling&quot;</td><td>14222.912109</td></tr><tr><td>&hellip;</td><td>&hellip;</td><td>&hellip;</td></tr><tr><td>2023</td><td>&quot;cycling&quot;</td><td>443997.875</td></tr><tr><td>2023</td><td>&quot;ebikeride&quot;</td><td>78663.265625</td></tr><tr><td>2017</td><td>&quot;cycling&quot;</td><td>5572.508301</td></tr><tr><td>2020</td><td>&quot;cycling&quot;</td><td>668791.9375</td></tr><tr><td>2024</td><td>&quot;ebikeride&quot;</td><td>1.3035e6</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (11, 3)\n",
"┌──────┬───────────┬──────────────┐\n",
"│ year ┆ type ┆ distance │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ i32 ┆ cat ┆ f32 │\n",
"╞══════╪═══════════╪══════════════╡\n",
"│ 2022 ┆ cycling ┆ 479097.40625 │\n",
"│ 2018 ┆ cycling ┆ 5883.088379 │\n",
"│ 2016 ┆ cycling ┆ 26981.732422 │\n",
"│ 2021 ┆ cycling ┆ 413450.9375 │\n",
"│ 2019 ┆ cycling ┆ 14222.912109 │\n",
"│ … ┆ … ┆ … │\n",
"│ 2023 ┆ cycling ┆ 443997.875 │\n",
"│ 2023 ┆ ebikeride ┆ 78663.265625 │\n",
"│ 2017 ┆ cycling ┆ 5572.508301 │\n",
"│ 2020 ┆ cycling ┆ 668791.9375 │\n",
"│ 2024 ┆ ebikeride ┆ 1.3035e6 │\n",
"└──────┴───────────┴──────────────┘"
]
},
"execution_count": 60,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# how far have I biked each year?\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .collect()\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "i73EK07Fb0Qc",
"outputId": "481579bf-ffe4-45bf-f7f5-af0d67ac7377"
},
"outputs": [
{
"ename": "AttributeError",
"evalue": "'LazyFrame' object has no attribute 'pivot'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[61], line 8\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# pivot type\u001b[39;00m\n\u001b[1;32m 2\u001b[0m (\u001b[43mtweak_strava\u001b[49m\u001b[43m(\u001b[49m\u001b[43mraw_lazy\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 3\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfilter\u001b[49m\u001b[43m(\u001b[49m\u001b[43mpl\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcol\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mtype\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mis_in\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mebikeride\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mcycling\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 4\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mgroup_by\u001b[49m\u001b[43m(\u001b[49m\u001b[43m[\u001b[49m\u001b[43mpl\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcol\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mtime\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdt\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43myear\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43malias\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43myear\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mtype\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 5\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43magg\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 6\u001b[0m \u001b[43m \u001b[49m\u001b[43mdistance\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mpl\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcol\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mdistance_2d\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msum\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 7\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m----> 8\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpivot\u001b[49m(index\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124myear\u001b[39m\u001b[38;5;124m'\u001b[39m, on\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtype\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 9\u001b[0m \u001b[38;5;241m.\u001b[39mcollect()\n\u001b[1;32m 10\u001b[0m )\n",
"\u001b[0;31mAttributeError\u001b[0m: 'LazyFrame' object has no attribute 'pivot'"
]
}
],
"source": [
"# pivot type\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .pivot(index='year', on='type')\n",
" .collect()\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "SdB_aKWib0Qc",
"outputId": "617ac546-92dc-473c-ac6f-b8c0cca1b9ae"
},
"outputs": [
{
"data": {
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>year</th><th>cycling</th><th>ebikeride</th></tr><tr><td>i32</td><td>f32</td><td>f32</td></tr></thead><tbody><tr><td>2021</td><td>413450.9375</td><td>null</td></tr><tr><td>2016</td><td>26981.732422</td><td>null</td></tr><tr><td>2023</td><td>443997.875</td><td>78663.265625</td></tr><tr><td>2019</td><td>14222.912109</td><td>null</td></tr><tr><td>2024</td><td>117095.242188</td><td>1.3035e6</td></tr><tr><td>2017</td><td>5572.508301</td><td>null</td></tr><tr><td>2022</td><td>479097.40625</td><td>null</td></tr><tr><td>2018</td><td>5883.088379</td><td>null</td></tr><tr><td>2020</td><td>668791.9375</td><td>null</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (9, 3)\n",
"┌──────┬───────────────┬──────────────┐\n",
"│ year ┆ cycling ┆ ebikeride │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ i32 ┆ f32 ┆ f32 │\n",
"╞══════╪═══════════════╪══════════════╡\n",
"│ 2021 ┆ 413450.9375 ┆ null │\n",
"│ 2016 ┆ 26981.732422 ┆ null │\n",
"│ 2023 ┆ 443997.875 ┆ 78663.265625 │\n",
"│ 2019 ┆ 14222.912109 ┆ null │\n",
"│ 2024 ┆ 117095.242188 ┆ 1.3035e6 │\n",
"│ 2017 ┆ 5572.508301 ┆ null │\n",
"│ 2022 ┆ 479097.40625 ┆ null │\n",
"│ 2018 ┆ 5883.088379 ┆ null │\n",
"│ 2020 ┆ 668791.9375 ┆ null │\n",
"└──────┴───────────────┴──────────────┘"
]
},
"execution_count": 63,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# pivot type\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .collect()\n",
" .pivot(index='year', on='type')\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 415
},
"id": "2C8CFXYvb0Qc",
"outputId": "d18c89ee-9fea-447c-e9ff-fa1406bfa449"
},
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 572 ms, sys: 69.9 ms, total: 641 ms\n",
"Wall time: 122 ms\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (9, 3)\n",
"┌──────┬──────────────┬───────────────┐\n",
"│ year ┆ ebikeride ┆ cycling │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ i32 ┆ f32 ┆ f32 │\n",
"╞══════╪══════════════╪═══════════════╡\n",
"│ 2016 ┆ null ┆ 26981.732422 │\n",
"│ 2017 ┆ null ┆ 5572.508301 │\n",
"│ 2018 ┆ null ┆ 5883.088379 │\n",
"│ 2019 ┆ null ┆ 14222.912109 │\n",
"│ 2020 ┆ null ┆ 668791.9375 │\n",
"│ 2021 ┆ null ┆ 413450.9375 │\n",
"│ 2022 ┆ null ┆ 479097.40625 │\n",
"│ 2023 ┆ 78663.265625 ┆ 443997.875 │\n",
"│ 2024 ┆ 1.3035e6 ┆ 117095.242188 │\n",
"└──────┴──────────────┴───────────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>year</th><th>ebikeride</th><th>cycling</th></tr><tr><td>i32</td><td>f32</td><td>f32</td></tr></thead><tbody><tr><td>2016</td><td>null</td><td>26981.732422</td></tr><tr><td>2017</td><td>null</td><td>5572.508301</td></tr><tr><td>2018</td><td>null</td><td>5883.088379</td></tr><tr><td>2019</td><td>null</td><td>14222.912109</td></tr><tr><td>2020</td><td>null</td><td>668791.9375</td></tr><tr><td>2021</td><td>null</td><td>413450.9375</td></tr><tr><td>2022</td><td>null</td><td>479097.40625</td></tr><tr><td>2023</td><td>78663.265625</td><td>443997.875</td></tr><tr><td>2024</td><td>1.3035e6</td><td>117095.242188</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 81
}
],
"source": [
"%%time\n",
"# pivot type\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .collect()\n",
" .pivot(index='year', on='type')\n",
" .sort('year')\n",
" )\n"
]
},
{
"cell_type": "code",
"source": [
"%%time\n",
"# pivot type\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .collect(engine='gpu')\n",
" # .collect(engine=pl.GPUEngine(device=0, raise_on_fail=True))\n",
" .pivot(index='year', on='type')\n",
" .sort('year')\n",
" )\n"
],
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 415
},
"id": "wWAHGiLBecXo",
"outputId": "d62ff9c1-7e26-4890-8bc7-f471b25dca82"
},
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"name": "stdout",
"text": [
"CPU times: user 117 ms, sys: 262 ms, total: 379 ms\n",
"Wall time: 804 ms\n"
]
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"shape: (9, 3)\n",
"┌──────┬──────────────┬───────────┐\n",
"│ year ┆ cycling ┆ ebikeride │\n",
"│ --- ┆ --- ┆ --- │\n",
"│ i32 ┆ f32 ┆ f32 │\n",
"╞══════╪══════════════╪═══════════╡\n",
"│ 2016 ┆ 26981.78125 ┆ null │\n",
"│ 2017 ┆ 5572.506348 ┆ null │\n",
"│ 2018 ┆ 5883.085449 ┆ null │\n",
"│ 2019 ┆ 14222.916016 ┆ null │\n",
"│ 2020 ┆ 668772.0625 ┆ null │\n",
"│ 2021 ┆ 413416.125 ┆ null │\n",
"│ 2022 ┆ 479068.78125 ┆ null │\n",
"│ 2023 ┆ 443981.0625 ┆ 78663.5 │\n",
"│ 2024 ┆ 117093.625 ┆ 1.3035e6 │\n",
"└──────┴──────────────┴───────────┘"
],
"text/html": [
"<div><style>\n",
".dataframe > thead > tr,\n",
".dataframe > tbody > tr {\n",
" text-align: right;\n",
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (9, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>year</th><th>cycling</th><th>ebikeride</th></tr><tr><td>i32</td><td>f32</td><td>f32</td></tr></thead><tbody><tr><td>2016</td><td>26981.78125</td><td>null</td></tr><tr><td>2017</td><td>5572.506348</td><td>null</td></tr><tr><td>2018</td><td>5883.085449</td><td>null</td></tr><tr><td>2019</td><td>14222.916016</td><td>null</td></tr><tr><td>2020</td><td>668772.0625</td><td>null</td></tr><tr><td>2021</td><td>413416.125</td><td>null</td></tr><tr><td>2022</td><td>479068.78125</td><td>null</td></tr><tr><td>2023</td><td>443981.0625</td><td>78663.5</td></tr><tr><td>2024</td><td>117093.625</td><td>1.3035e6</td></tr></tbody></table></div>"
]
},
"metadata": {},
"execution_count": 82
}
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "RMkLifAmb0Qd",
"outputId": "91989f6d-71e1-4a61-a616-2a47869579ff"
},
"outputs": [
{
"data": {},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.holoviews_exec.v0+json": "",
"text/html": [
"<div id='p1076'>\n",
" <div id=\"d6200bb1-f43c-4079-8514-00a43fd0d25a\" data-root-id=\"p1076\" style=\"display: contents;\"></div>\n",
"</div>\n",
"<script type=\"application/javascript\">(function(root) {\n",
" var docs_json = {\"5bcac29e-70af-404a-ad77-f81d15a9ca95\":{\"version\":\"3.3.4\",\"title\":\"Bokeh Application\",\"roots\":[{\"type\":\"object\",\"name\":\"Row\",\"id\":\"p1076\",\"attributes\":{\"name\":\"Row01321\",\"tags\":[\"embedded\"],\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1079\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/loading.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1154\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/css/listpanel.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1077\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/default.css\"}},{\"type\":\"object\",\"name\":\"ImportedStyleSheet\",\"id\":\"p1078\",\"attributes\":{\"url\":\"https://cdn.holoviz.org/panel/1.3.8/dist/bundled/theme/native.css\"}}],\"min_width\":700,\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\",\"children\":[{\"type\":\"object\",\"name\":\"Spacer\",\"id\":\"p1080\",\"attributes\":{\"name\":\"HSpacer01328\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"p1079\"},{\"id\":\"p1077\"},{\"id\":\"p1078\"}],\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\"}},{\"type\":\"object\",\"name\":\"Figure\",\"id\":\"p1091\",\"attributes\":{\"width\":700,\"height\":300,\"margin\":[5,10],\"sizing_mode\":\"fixed\",\"align\":\"start\",\"x_range\":{\"type\":\"object\",\"name\":\"Range1d\",\"id\":\"p1081\",\"attributes\":{\"tags\":[[[\"year\",\"year\",null]],[]],\"start\":2016.0,\"end\":2024.0,\"reset_start\":2016.0,\"reset_end\":2024.0}},\"y_range\":{\"type\":\"object\",\"name\":\"Range1d\",\"id\":\"p1082\",\"attributes\":{\"tags\":[[[\"value\",\"value\",null]],{\"type\":\"map\",\"entries\":[[\"invert_yaxis\",false],[\"autorange\",false]]}],\"start\":-124224.37836914063,\"end\":1433338.261669922,\"reset_start\":-124224.37836914063,\"reset_end\":1433338.261669922}},\"x_scale\":{\"type\":\"object\",\"name\":\"LinearScale\",\"id\":\"p1101\"},\"y_scale\":{\"type\":\"object\",\"name\":\"LinearScale\",\"id\":\"p1102\"},\"title\":{\"type\":\"object\",\"name\":\"Title\",\"id\":\"p1094\",\"attributes\":{\"text_color\":\"black\",\"text_font_size\":\"12pt\"}},\"renderers\":[{\"type\":\"object\",\"name\":\"GlyphRenderer\",\"id\":\"p1131\",\"attributes\":{\"name\":\"cycling\",\"data_source\":{\"type\":\"object\",\"name\":\"ColumnDataSource\",\"id\":\"p1122\",\"attributes\":{\"selected\":{\"type\":\"object\",\"name\":\"Selection\",\"id\":\"p1123\",\"attributes\":{\"indices\":[],\"line_indices\":[]}},\"selection_policy\":{\"type\":\"object\",\"name\":\"UnionRenderers\",\"id\":\"p1124\"},\"data\":{\"type\":\"map\",\"entries\":[[\"year\",{\"type\":\"ndarray\",\"array\":{\"type\":\"bytes\",\"data\":\"4AcAAOEHAADiBwAA4wcAAOQHAADlBwAA5gcAAOcHAADoBwAA\"},\"shape\":[9],\"dtype\":\"int32\",\"order\":\"little\"}],[\"value\",{\"type\":\"ndarray\",\"array\":{\"type\":\"bytes\",\"data\":\"d8vSRhEkrkW12LdFpjteRn9HI0le4clILe/pSLzL2Eifs+RH\"},\"shape\":[9],\"dtype\":\"float32\",\"order\":\"little\"}],[\"Variable\",[\"cycling\",\"cycling\",\"cycling\",\"cycling\",\"cycling\",\"cycling\",\"cycling\",\"cycling\",\"cycling\"]]]}}},\"view\":{\"type\":\"object\",\"name\":\"CDSView\",\"id\":\"p1132\",\"attributes\":{\"filter\":{\"type\":\"object\",\"name\":\"AllIndices\",\"id\":\"p1133\"}}},\"glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1128\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#30a2da\",\"line_width\":2}},\"selection_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1136\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#30a2da\",\"line_width\":2}},\"nonselection_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1129\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#30a2da\",\"line_alpha\":0.1,\"line_width\":2}},\"muted_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1130\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#30a2da\",\"line_alpha\":0.2,\"line_width\":2}}}},{\"type\":\"object\",\"name\":\"GlyphRenderer\",\"id\":\"p1146\",\"attributes\":{\"name\":\"ebikeride\",\"data_source\":{\"type\":\"object\",\"name\":\"ColumnDataSource\",\"id\":\"p1137\",\"attributes\":{\"selected\":{\"type\":\"object\",\"name\":\"Selection\",\"id\":\"p1138\",\"attributes\":{\"indices\":[],\"line_indices\":[]}},\"selection_policy\":{\"type\":\"object\",\"name\":\"UnionRenderers\",\"id\":\"p1139\"},\"data\":{\"type\":\"map\",\"entries\":[[\"year\",{\"type\":\"ndarray\",\"array\":{\"type\":\"bytes\",\"data\":\"4AcAAOEHAADiBwAA4wcAAOQHAADlBwAA5gcAAOcHAADoBwAA\"},\"shape\":[9],\"dtype\":\"int32\",\"order\":\"little\"}],[\"value\",{\"type\":\"ndarray\",\"array\":{\"type\":\"bytes\",\"data\":\"AADAfwAAwH8AAMB/AADAfwAAwH8AAMB/AADAf6KjmUerH59J\"},\"shape\":[9],\"dtype\":\"float32\",\"order\":\"little\"}],[\"Variable\",[\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\",\"ebikeride\"]]]}}},\"view\":{\"type\":\"object\",\"name\":\"CDSView\",\"id\":\"p1147\",\"attributes\":{\"filter\":{\"type\":\"object\",\"name\":\"AllIndices\",\"id\":\"p1148\"}}},\"glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1143\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#fc4f30\",\"line_width\":2}},\"selection_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1150\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#fc4f30\",\"line_width\":2}},\"nonselection_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1144\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#fc4f30\",\"line_alpha\":0.1,\"line_width\":2}},\"muted_glyph\":{\"type\":\"object\",\"name\":\"Line\",\"id\":\"p1145\",\"attributes\":{\"tags\":[\"apply_ranges\"],\"x\":{\"type\":\"field\",\"field\":\"year\"},\"y\":{\"type\":\"field\",\"field\":\"value\"},\"line_color\":\"#fc4f30\",\"line_alpha\":0.2,\"line_width\":2}}}}],\"toolbar\":{\"type\":\"object\",\"name\":\"Toolbar\",\"id\":\"p1100\",\"attributes\":{\"tools\":[{\"type\":\"object\",\"name\":\"WheelZoomTool\",\"id\":\"p1086\",\"attributes\":{\"tags\":[\"hv_created\"],\"renderers\":\"auto\",\"zoom_together\":\"none\"}},{\"type\":\"object\",\"name\":\"HoverTool\",\"id\":\"p1087\",\"attributes\":{\"tags\":[\"hv_created\"],\"renderers\":[{\"id\":\"p1131\"},{\"id\":\"p1146\"}],\"tooltips\":[[\"Variable\",\"@{Variable}\"],[\"year\",\"@{year}\"],[\"value\",\"@{value}\"]]}},{\"type\":\"object\",\"name\":\"SaveTool\",\"id\":\"p1113\"},{\"type\":\"object\",\"name\":\"PanTool\",\"id\":\"p1114\"},{\"type\":\"object\",\"name\":\"BoxZoomTool\",\"id\":\"p1115\",\"attributes\":{\"overlay\":{\"type\":\"object\",\"name\":\"BoxAnnotation\",\"id\":\"p1116\",\"attributes\":{\"syncable\":false,\"level\":\"overlay\",\"visible\":false,\"left\":{\"type\":\"number\",\"value\":\"nan\"},\"right\":{\"type\":\"number\",\"value\":\"nan\"},\"top\":{\"type\":\"number\",\"value\":\"nan\"},\"bottom\":{\"type\":\"number\",\"value\":\"nan\"},\"left_units\":\"canvas\",\"right_units\":\"canvas\",\"top_units\":\"canvas\",\"bottom_units\":\"canvas\",\"line_color\":\"black\",\"line_alpha\":1.0,\"line_width\":2,\"line_dash\":[4,4],\"fill_color\":\"lightgrey\",\"fill_alpha\":0.5}}}},{\"type\":\"object\",\"name\":\"ResetTool\",\"id\":\"p1121\"}],\"active_drag\":{\"id\":\"p1114\"},\"active_scroll\":{\"id\":\"p1086\"}}},\"left\":[{\"type\":\"object\",\"name\":\"LinearAxis\",\"id\":\"p1108\",\"attributes\":{\"ticker\":{\"type\":\"object\",\"name\":\"BasicTicker\",\"id\":\"p1109\",\"attributes\":{\"mantissas\":[1,2,5]}},\"formatter\":{\"type\":\"object\",\"name\":\"BasicTickFormatter\",\"id\":\"p1110\"},\"axis_label\":\"\",\"major_label_policy\":{\"type\":\"object\",\"name\":\"AllLabels\",\"id\":\"p1111\"}}}],\"right\":[{\"type\":\"object\",\"name\":\"Legend\",\"id\":\"p1134\",\"attributes\":{\"location\":[0,0],\"title\":\"Variable\",\"click_policy\":\"mute\",\"items\":[{\"type\":\"object\",\"name\":\"LegendItem\",\"id\":\"p1135\",\"attributes\":{\"label\":{\"type\":\"value\",\"value\":\"cycling\"},\"renderers\":[{\"id\":\"p1131\"}]}},{\"type\":\"object\",\"name\":\"LegendItem\",\"id\":\"p1149\",\"attributes\":{\"label\":{\"type\":\"value\",\"value\":\"ebikeride\"},\"renderers\":[{\"id\":\"p1146\"}]}}]}}],\"below\":[{\"type\":\"object\",\"name\":\"LinearAxis\",\"id\":\"p1103\",\"attributes\":{\"ticker\":{\"type\":\"object\",\"name\":\"BasicTicker\",\"id\":\"p1104\",\"attributes\":{\"mantissas\":[1,2,5]}},\"formatter\":{\"type\":\"object\",\"name\":\"BasicTickFormatter\",\"id\":\"p1105\"},\"axis_label\":\"year\",\"major_label_policy\":{\"type\":\"object\",\"name\":\"AllLabels\",\"id\":\"p1106\"}}}],\"center\":[{\"type\":\"object\",\"name\":\"Grid\",\"id\":\"p1107\",\"attributes\":{\"axis\":{\"id\":\"p1103\"},\"grid_line_color\":null}},{\"type\":\"object\",\"name\":\"Grid\",\"id\":\"p1112\",\"attributes\":{\"dimension\":1,\"axis\":{\"id\":\"p1108\"},\"grid_line_color\":null}}],\"min_border_top\":10,\"min_border_bottom\":10,\"min_border_left\":10,\"min_border_right\":10,\"output_backend\":\"webgl\"}},{\"type\":\"object\",\"name\":\"Spacer\",\"id\":\"p1152\",\"attributes\":{\"name\":\"HSpacer01329\",\"stylesheets\":[\"\\n:host(.pn-loading.pn-arc):before, .pn-loading.pn-arc:before {\\n background-image: url(\\\"data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHN0eWxlPSJtYXJnaW46IGF1dG87IGJhY2tncm91bmQ6IG5vbmU7IGRpc3BsYXk6IGJsb2NrOyBzaGFwZS1yZW5kZXJpbmc6IGF1dG87IiB2aWV3Qm94PSIwIDAgMTAwIDEwMCIgcHJlc2VydmVBc3BlY3RSYXRpbz0ieE1pZFlNaWQiPiAgPGNpcmNsZSBjeD0iNTAiIGN5PSI1MCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSIjYzNjM2MzIiBzdHJva2Utd2lkdGg9IjEwIiByPSIzNSIgc3Ryb2tlLWRhc2hhcnJheT0iMTY0LjkzMzYxNDMxMzQ2NDE1IDU2Ljk3Nzg3MTQzNzgyMTM4Ij4gICAgPGFuaW1hdGVUcmFuc2Zvcm0gYXR0cmlidXRlTmFtZT0idHJhbnNmb3JtIiB0eXBlPSJyb3RhdGUiIHJlcGVhdENvdW50PSJpbmRlZmluaXRlIiBkdXI9IjFzIiB2YWx1ZXM9IjAgNTAgNTA7MzYwIDUwIDUwIiBrZXlUaW1lcz0iMDsxIj48L2FuaW1hdGVUcmFuc2Zvcm0+ICA8L2NpcmNsZT48L3N2Zz4=\\\");\\n background-size: auto calc(min(50%, 400px));\\n}\",{\"id\":\"p1079\"},{\"id\":\"p1077\"},{\"id\":\"p1078\"}],\"margin\":0,\"sizing_mode\":\"stretch_width\",\"align\":\"start\"}}]}}],\"defs\":[{\"type\":\"model\",\"name\":\"ReactiveHTML1\"},{\"type\":\"model\",\"name\":\"FlexBox1\",\"properties\":[{\"name\":\"align_content\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"align_items\",\"kind\":\"Any\",\"default\":\"flex-start\"},{\"name\":\"flex_direction\",\"kind\":\"Any\",\"default\":\"row\"},{\"name\":\"flex_wrap\",\"kind\":\"Any\",\"default\":\"wrap\"},{\"name\":\"justify_content\",\"kind\":\"Any\",\"default\":\"flex-start\"}]},{\"type\":\"model\",\"name\":\"FloatPanel1\",\"properties\":[{\"name\":\"config\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"contained\",\"kind\":\"Any\",\"default\":true},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"right-top\"},{\"name\":\"offsetx\",\"kind\":\"Any\",\"default\":null},{\"name\":\"offsety\",\"kind\":\"Any\",\"default\":null},{\"name\":\"theme\",\"kind\":\"Any\",\"default\":\"primary\"},{\"name\":\"status\",\"kind\":\"Any\",\"default\":\"normalized\"}]},{\"type\":\"model\",\"name\":\"GridStack1\",\"properties\":[{\"name\":\"mode\",\"kind\":\"Any\",\"default\":\"warn\"},{\"name\":\"ncols\",\"kind\":\"Any\",\"default\":null},{\"name\":\"nrows\",\"kind\":\"Any\",\"default\":null},{\"name\":\"allow_resize\",\"kind\":\"Any\",\"default\":true},{\"name\":\"allow_drag\",\"kind\":\"Any\",\"default\":true},{\"name\":\"state\",\"kind\":\"Any\",\"default\":[]}]},{\"type\":\"model\",\"name\":\"drag1\",\"properties\":[{\"name\":\"slider_width\",\"kind\":\"Any\",\"default\":5},{\"name\":\"slider_color\",\"kind\":\"Any\",\"default\":\"black\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":50}]},{\"type\":\"model\",\"name\":\"click1\",\"properties\":[{\"name\":\"terminal_output\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"debug_name\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"clears\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"copy_to_clipboard1\",\"properties\":[{\"name\":\"fill\",\"kind\":\"Any\",\"default\":\"none\"},{\"name\":\"value\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"FastWrapper1\",\"properties\":[{\"name\":\"object\",\"kind\":\"Any\",\"default\":null},{\"name\":\"style\",\"kind\":\"Any\",\"default\":null}]},{\"type\":\"model\",\"name\":\"NotificationAreaBase1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"NotificationArea1\",\"properties\":[{\"name\":\"js_events\",\"kind\":\"Any\",\"default\":{\"type\":\"map\"}},{\"name\":\"notifications\",\"kind\":\"Any\",\"default\":[]},{\"name\":\"position\",\"kind\":\"Any\",\"default\":\"bottom-right\"},{\"name\":\"_clear\",\"kind\":\"Any\",\"default\":0},{\"name\":\"types\",\"kind\":\"Any\",\"default\":[{\"type\":\"map\",\"entries\":[[\"type\",\"warning\"],[\"background\",\"#ffc107\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-exclamation-triangle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]},{\"type\":\"map\",\"entries\":[[\"type\",\"info\"],[\"background\",\"#007bff\"],[\"icon\",{\"type\":\"map\",\"entries\":[[\"className\",\"fas fa-info-circle\"],[\"tagName\",\"i\"],[\"color\",\"white\"]]}]]}]}]},{\"type\":\"model\",\"name\":\"Notification\",\"properties\":[{\"name\":\"background\",\"kind\":\"Any\",\"default\":null},{\"name\":\"duration\",\"kind\":\"Any\",\"default\":3000},{\"name\":\"icon\",\"kind\":\"Any\",\"default\":null},{\"name\":\"message\",\"kind\":\"Any\",\"default\":\"\"},{\"name\":\"notification_type\",\"kind\":\"Any\",\"default\":null},{\"name\":\"_destroyed\",\"kind\":\"Any\",\"default\":false}]},{\"type\":\"model\",\"name\":\"TemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"BootstrapTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]},{\"type\":\"model\",\"name\":\"MaterialTemplateActions1\",\"properties\":[{\"name\":\"open_modal\",\"kind\":\"Any\",\"default\":0},{\"name\":\"close_modal\",\"kind\":\"Any\",\"default\":0}]}]}};\n",
" var render_items = [{\"docid\":\"5bcac29e-70af-404a-ad77-f81d15a9ca95\",\"roots\":{\"p1076\":\"d6200bb1-f43c-4079-8514-00a43fd0d25a\"},\"root_ids\":[\"p1076\"]}];\n",
" var docs = Object.values(docs_json)\n",
" if (!docs) {\n",
" return\n",
" }\n",
" const py_version = docs[0].version.replace('rc', '-rc.').replace('.dev', '-dev.')\n",
" function embed_document(root) {\n",
" var Bokeh = get_bokeh(root)\n",
" Bokeh.embed.embed_items_notebook(docs_json, render_items);\n",
" for (const render_item of render_items) {\n",
" for (const root_id of render_item.root_ids) {\n",
"\tconst id_el = document.getElementById(root_id)\n",
"\tif (id_el.children.length && (id_el.children[0].className === 'bk-root')) {\n",
"\t const root_el = id_el.children[0]\n",
"\t root_el.id = root_el.id + '-rendered'\n",
"\t}\n",
" }\n",
" }\n",
" }\n",
" function get_bokeh(root) {\n",
" if (root.Bokeh === undefined) {\n",
" return null\n",
" } else if (root.Bokeh.version !== py_version) {\n",
" if (root.Bokeh.versions === undefined || !root.Bokeh.versions.has(py_version)) {\n",
"\treturn null\n",
" }\n",
" return root.Bokeh.versions.get(py_version);\n",
" } else if (root.Bokeh.version === py_version) {\n",
" return root.Bokeh\n",
" }\n",
" return null\n",
" }\n",
" function is_loaded(root) {\n",
" var Bokeh = get_bokeh(root)\n",
" return (Bokeh != null && Bokeh.Panel !== undefined)\n",
" }\n",
" if (is_loaded(root)) {\n",
" embed_document(root);\n",
" } else {\n",
" var attempts = 0;\n",
" var timer = setInterval(function(root) {\n",
" if (is_loaded(root)) {\n",
" clearInterval(timer);\n",
" embed_document(root);\n",
" } else if (document.readyState == \"complete\") {\n",
" attempts++;\n",
" if (attempts > 200) {\n",
" clearInterval(timer);\n",
"\t var Bokeh = get_bokeh(root)\n",
"\t if (Bokeh == null || Bokeh.Panel == null) {\n",
" console.warn(\"Panel: ERROR: Unable to run Panel code because Bokeh or Panel library is missing\");\n",
"\t } else {\n",
"\t console.warn(\"Panel: WARNING: Attempting to render but not all required libraries could be resolved.\")\n",
"\t embed_document(root)\n",
"\t }\n",
" }\n",
" }\n",
" }, 25, root)\n",
" }\n",
"})(window);</script>"
],
"text/plain": [
":NdOverlay [Variable]\n",
" :Curve [year] (value)"
]
},
"execution_count": 72,
"metadata": {
"application/vnd.holoviews_exec.v0+json": {
"id": "p1076"
}
},
"output_type": "execute_result"
}
],
"source": [
"# pivot type\n",
"(tweak_strava(raw_lazy)\n",
" .filter(pl.col('type').is_in(['ebikeride', 'cycling']))\n",
" .group_by([pl.col('time').dt.year().alias('year'), 'type'])\n",
" .agg(\n",
" distance=pl.col('distance_2d').sum(),\n",
" )\n",
" .collect()\n",
" .pivot(index='year', on='type')\n",
" .sort('year')\n",
" .hvplot.line(x='year', y=['cycling', 'ebikeride'])\n",
" )\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "WnV2Gha4b0Qd"
},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "O6Z9Kh_jb0Qd"
},
"source": [
"## Conclusion\n",
"\n",
"* Correct types save space and enable convenient math, string, and date functionality\n",
"* Chaining operations will:\n",
" * Make code readable\n",
" * Remove bugs\n",
" * Easier to debug\n",
"* ``.map_elements`` is slow for math\n",
"* Aggregations are powerful. Play with them until they make sense\n",
"\n",
"\n",
"Let's connect! Reach out (email, LinkedIn) if your team needs help with the Python data stack.\n",
"\n",
"Book giveaway"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "Bxam8iwib0Qd"
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.16"
},
"colab": {
"provenance": [],
"toc_visible": true,
"machine_shape": "hm",
"gpuType": "L4",
"include_colab_link": true
},
"accelerator": "GPU"
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment