Skip to content

Instantly share code, notes, and snippets.

@ahmadia
Forked from bfroehle/.gitignore
Created May 17, 2013 11:23

Revisions

  1. @bfroehle bfroehle revised this gist Dec 7, 2012. 1 changed file with 7 additions and 1 deletion.
    8 changes: 7 additions & 1 deletion biteymagic.py
    Original file line number Diff line number Diff line change
    @@ -75,10 +75,16 @@ def bitey(self, line, cell):

    if not os.path.exists(o_path):
    try:
    startupinfo = None
    if os.name == 'nt':
    # Avoid a console window in Microsoft Windows.
    startupinfo = subprocess.STARTUPINFO()
    startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW
    subprocess.check_output(['clang', '-c', '-emit-llvm'] +
    line.split() + [c_name],
    stderr=subprocess.STDOUT,
    cwd=lib_dir)
    cwd=lib_dir,
    startupinfo=startupinfo)
    except subprocess.CalledProcessError as e:
    print(e.output, file=sys.stderr)
    print("ERROR: command `%s` failed." %
  2. @bfroehle bfroehle revised this gist Dec 7, 2012. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions .gitignore
    Original file line number Diff line number Diff line change
    @@ -0,0 +1 @@
    *.py[oc]
  3. @bfroehle bfroehle revised this gist Aug 27, 2012. 1 changed file with 202 additions and 48 deletions.
    250 changes: 202 additions & 48 deletions Bitey Magic.ipynb
    Original file line number Diff line number Diff line change
    @@ -7,6 +7,26 @@
    "worksheets": [
    {
    "cells": [
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Bitey Magic\n",
    "===========\n",
    "\n",
    "[Bitey] is a tool to import LLVM bitcode directly into Python. Bitey magic is a small [IPython] extension which adds a `%%bitey` cell magic for automatically compiling C (or C++) code into LLVM bitcode and loading the bitcode with Bitey.\n",
    "\n",
    "Requirements\n",
    "------------\n",
    "\n",
    "* [Bitey], importable from Python as `import bitey`\n",
    "* [clang], executable as `clang` in the shell\n",
    "\n",
    "[Bitey]: https://github.com/dabeaz/bitey\n",
    "[IPython]: http://www.ipython.org/\n",
    "[clang]: http://clang.llvm.org/"
    ]
    },
    {
    "cell_type": "code",
    "collapsed": false,
    @@ -16,7 +36,27 @@
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 1
    "prompt_number": 17
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Demos\n",
    "=====\n",
    "\n",
    "The following demos are taken, with slight modification, from the examples in the Bitey source."
    ]
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "The Basics\n",
    "----------\n",
    "\n",
    "Calculate the Fibonacci numbers using a simple recursive formula."
    ]
    },
    {
    "cell_type": "code",
    @@ -34,26 +74,88 @@
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 2
    "prompt_number": 18
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "map(fib, xrange(10))"
    "map(fib, xrange(1,10))"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "pyout",
    "prompt_number": 3,
    "prompt_number": 20,
    "text": [
    "[1, 1, 1, 2, 3, 5, 8, 13, 21, 34]"
    "[1, 1, 2, 3, 5, 8, 13, 21, 34]"
    ]
    }
    ],
    "prompt_number": 3
    "prompt_number": 20
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Bitey understands most basic C datatypes including integers, floats,\n",
    "void, pointers, arrays, and structures. Because it builds a ctypes\n",
    "based interface, you would access the code using the same\n",
    "techniques. Here is an example that mutates a value through a\n",
    "pointer."
    ]
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey\n",
    "void mutate_int(int *x) {\n",
    " *x *= 2;\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 9
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "import ctypes\n",
    "x = ctypes.c_int(2)\n",
    "mutate_int(x)\n",
    "x.value"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "pyout",
    "prompt_number": 10,
    "text": [
    "4"
    ]
    }
    ],
    "prompt_number": 10
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Structs\n",
    "-------\n",
    "\n",
    "An example involing a structure. One subtle issue with structure wrapping is that LLVM bitcode doesn't encode the names of structure fields. So, Bitey simply assigns them to an indexed element variable like this:\n",
    "\n",
    " >>> p1.e0 # (Returns the .x component)\n",
    " 3\n",
    " >>> p1.e1 # (Returns the .y component)\n",
    " 4\n"
    ]
    },
    {
    "cell_type": "code",
    @@ -100,11 +202,21 @@
    ],
    "prompt_number": 5
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Performance\n",
    "-----------\n",
    "\n",
    "The performance profile of Bitey is going to be virtually identical that of using ``ctypes``. LLVM bitcode is translated to native machine code and Bitey builds a ``ctypes``-based interface to it in exactly the same manner as a normal C library."
    ]
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey\n",
    "%%bitey -03\n",
    "/* A function that determines if an integer is a prime number or not.\n",
    " This is just a naive implementation--there are faster ways to do it */\n",
    "\n",
    @@ -131,7 +243,7 @@
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 6
    "prompt_number": 7
    },
    {
    "cell_type": "code",
    @@ -147,18 +259,25 @@
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "10000 loops, best of 3: 37.1 us per loop\n"
    "10000 loops, best of 3: 36.8 us per loop\n"
    ]
    }
    ],
    "prompt_number": 7
    "prompt_number": 8
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "Additional optimization flags given after `%%bitey` are passed directly to `clang`. For example, consider a simple function which sums a vector."
    ]
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey -O2\n",
    "double sum1d_c(double* x, int n)\n",
    "%%bitey\n",
    "double sum1d(double* x, int n)\n",
    "{\n",
    " int i;\n",
    " double ret;\n",
    @@ -171,50 +290,40 @@
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 8
    "prompt_number": 11
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey -O2 -x c++\n",
    "#include <numeric>\n",
    "extern \"C\" double sum1d_cpp(double* x, int n)\n",
    "%%bitey -O2\n",
    "double sum1d_O2(double* x, int n)\n",
    "{\n",
    " return std::accumulate(x, x+n, 0.0);\n",
    " int i;\n",
    " double ret;\n",
    " ret = 0.0;\n",
    " for(i=0; i<n; i++)\n",
    " ret = ret + x[i];\n",
    " return ret;\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 9
    "prompt_number": 12
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "import numpy\n",
    "n = 10000\n",
    "A = numpy.random.rand(n)\n",
    "x = numpy.ctypeslib.as_ctypes(A)"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 10
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "print('A.sum:')\n",
    "%timeit A.sum()\n",
    "\n",
    "print('sum1d_c:')\n",
    "%timeit sum1d_c(x, n)\n",
    "\n",
    "print('sum1d_cpp:')\n",
    "%timeit sum1d_cpp(x, n)"
    "import numpy as np\n",
    "n = 1000\n",
    "A = np.random.rand(n)\n",
    "x = np.ctypeslib.as_ctypes(A)\n",
    "print('sum1d:')\n",
    "%timeit sum1d(x, n)\n",
    "print('sum1d_O2:')\n",
    "%timeit sum1d_O2(x, n)"
    ],
    "language": "python",
    "metadata": {},
    @@ -223,26 +332,71 @@
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "A.sum:\n",
    "100000 loops, best of 3: 19.7 us per loop"
    "sum1d:\n",
    "100000 loops, best of 3: 7.39 us per loop"
    ]
    },
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "\n",
    "sum1d_c:\n",
    "100000 loops, best of 3: 19.9 us per loop"
    "sum1d_O2:\n",
    "100000 loops, best of 3: 3.29 us per loop"
    ]
    },
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "\n",
    "sum1d_cpp:\n",
    "100000 loops, best of 3: 19.9 us per loop"
    "\n"
    ]
    }
    ],
    "prompt_number": 14
    },
    {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
    "C++ Support\n",
    "-----------\n",
    "\n",
    "Bitey does not support C++ functions, but C++ code can be compiled if the function is decorated with `extern \"C\"`. In addition, you will need to pass `-x c++` to `clang` to specify that the language is C++."
    ]
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey -O2 -x c++\n",
    "#include <numeric>\n",
    "extern \"C\" double sum1d_cpp(double* x, int n)\n",
    "{\n",
    " return std::accumulate(x, x+n, 0.0);\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 15
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "print('sum1d_cpp')\n",
    "%timeit sum1d_cpp(x, n)"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "sum1d_cpp\n",
    "100000 loops, best of 3: 3.33 us per loop"
    ]
    },
    {
    @@ -253,7 +407,7 @@
    ]
    }
    ],
    "prompt_number": 11
    "prompt_number": 16
    }
    ],
    "metadata": {}
  4. @bfroehle bfroehle created this gist Aug 25, 2012.
    262 changes: 262 additions & 0 deletions Bitey Magic.ipynb
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,262 @@
    {
    "metadata": {
    "name": "Bitey Magic"
    },
    "nbformat": 3,
    "nbformat_minor": 0,
    "worksheets": [
    {
    "cells": [
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%load_ext biteymagic"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 1
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey\n",
    "int fib(int n) {\n",
    " if (n < 3) {\n",
    " return 1;\n",
    " } else {\n",
    " return fib(n-2) + fib(n-1);\n",
    " }\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 2
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "map(fib, xrange(10))"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "pyout",
    "prompt_number": 3,
    "text": [
    "[1, 1, 1, 2, 3, 5, 8, 13, 21, 34]"
    ]
    }
    ],
    "prompt_number": 3
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey\n",
    "#include <math.h>\n",
    "\n",
    "struct Point {\n",
    " double x;\n",
    " double y;\n",
    "};\n",
    "\n",
    "double distance(struct Point *p1, struct Point *p2) {\n",
    " return sqrt((p1->x - p2->x)*(p1->x - p2->x) + \n",
    "\t (p1->y - p2->y)*(p1->y - p2->y));\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 4
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "p1 = Point(3,4)\n",
    "p2 = Point(6,8)\n",
    "print distance(p1,p2)\n",
    "print p1.e0, p1.e1"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "5.0\n",
    "3.0 4.0\n"
    ]
    }
    ],
    "prompt_number": 5
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey\n",
    "/* A function that determines if an integer is a prime number or not.\n",
    " This is just a naive implementation--there are faster ways to do it */\n",
    "\n",
    "int isprime(int n) {\n",
    " int factor = 3;\n",
    " /* Special case for 2 */\n",
    " if (n == 2) {\n",
    " return 1;\n",
    " }\n",
    " /* Check for even numbers */\n",
    " if ((n % 2) == 0) {\n",
    " return 0;\n",
    " }\n",
    " /* Check for everything else */\n",
    " while (factor*factor < n) {\n",
    " if ((n % factor) == 0) {\n",
    " return 0;\n",
    " }\n",
    " factor += 2;\n",
    " }\n",
    " return 1;\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 6
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%timeit\n",
    "isprime(10143937)"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "10000 loops, best of 3: 37.1 us per loop\n"
    ]
    }
    ],
    "prompt_number": 7
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey -O2\n",
    "double sum1d_c(double* x, int n)\n",
    "{\n",
    " int i;\n",
    " double ret;\n",
    " ret = 0.0;\n",
    " for(i=0; i<n; i++)\n",
    " ret = ret + x[i];\n",
    " return ret;\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 8
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "%%bitey -O2 -x c++\n",
    "#include <numeric>\n",
    "extern \"C\" double sum1d_cpp(double* x, int n)\n",
    "{\n",
    " return std::accumulate(x, x+n, 0.0);\n",
    "}"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 9
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "import numpy\n",
    "n = 10000\n",
    "A = numpy.random.rand(n)\n",
    "x = numpy.ctypeslib.as_ctypes(A)"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [],
    "prompt_number": 10
    },
    {
    "cell_type": "code",
    "collapsed": false,
    "input": [
    "print('A.sum:')\n",
    "%timeit A.sum()\n",
    "\n",
    "print('sum1d_c:')\n",
    "%timeit sum1d_c(x, n)\n",
    "\n",
    "print('sum1d_cpp:')\n",
    "%timeit sum1d_cpp(x, n)"
    ],
    "language": "python",
    "metadata": {},
    "outputs": [
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "A.sum:\n",
    "100000 loops, best of 3: 19.7 us per loop"
    ]
    },
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "\n",
    "sum1d_c:\n",
    "100000 loops, best of 3: 19.9 us per loop"
    ]
    },
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "\n",
    "sum1d_cpp:\n",
    "100000 loops, best of 3: 19.9 us per loop"
    ]
    },
    {
    "output_type": "stream",
    "stream": "stdout",
    "text": [
    "\n"
    ]
    }
    ],
    "prompt_number": 11
    }
    ],
    "metadata": {}
    }
    ]
    }
    103 changes: 103 additions & 0 deletions biteymagic.py
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,103 @@
    # -*- coding: utf-8 -*-
    """Bitey related magics.
    Bitey and a list of its requirements may be found at
    https://github.com/dabeaz/bitey.
    """
    #-----------------------------------------------------------------------------
    # Copyright (C) 2012, IPython Development Team.
    #
    # Distributed under the terms of the Modified BSD License.
    #
    # The full license is in the file COPYING.txt, distributed with this software.
    #-----------------------------------------------------------------------------

    from __future__ import print_function

    import bitey
    import imp
    import io
    import os
    import pipes
    import subprocess
    import sys

    try:
    import hashlib
    except ImportError:
    import md5 as hashlib

    from IPython.core.magic import Magics, magics_class, cell_magic

    @magics_class
    class BiteyMagics(Magics):
    """Magics for Bitey, a Bitcode Import Tool"""

    def _import_all(self, module):
    for k,v in module.__dict__.items():
    if not k.startswith('__'):
    self.shell.push({k:v})

    @cell_magic
    def bitey(self, line, cell):
    """Compile C code into LLVM bitcode and import using bitey.
    Usage, in cell mode::
    %%bitey <compiler flags>
    <C code>
    The compiler flags are passed verbatim to `clang` so they may be
    used to control warnings (`-Wall`), add optimizations (`-O2`), and
    modify features (`-fno-builtin`).
    Bitey may also be used to compile C++ code if the functions are
    defined with C linkage (`extern "C"`) and `-x c++` is added to the
    compiler flags.
    """
    code = cell if cell.endswith('\n') else cell+'\n'
    lib_dir = os.path.join(self.shell.ipython_dir, 'bitey')
    key = line, code, sys.version_info, sys.executable
    if not os.path.exists(lib_dir):
    os.makedirs(lib_dir)

    module_name = "_bitey_magic_" + \
    hashlib.md5(str(key).encode('utf-8')).hexdigest()
    c_name = module_name+'.c'
    o_name = module_name+'.o'

    c_path = os.path.join(lib_dir, c_name)
    o_path = os.path.join(lib_dir, o_name)

    if not os.path.exists(c_path):
    with io.open(c_path, 'w', encoding='utf-8') as f:
    f.write(code)

    if not os.path.exists(o_path):
    try:
    subprocess.check_output(['clang', '-c', '-emit-llvm'] +
    line.split() + [c_name],
    stderr=subprocess.STDOUT,
    cwd=lib_dir)
    except subprocess.CalledProcessError as e:
    print(e.output, file=sys.stderr)
    print("ERROR: command `%s` failed." %
    ' '.join(map(pipes.quote, e.cmd)),
    file=sys.stderr)
    return

    with io.open(o_path, 'rb') as f:
    bitcode = f.read()

    module = bitey.loader.build_module(module_name, bitcode)
    sys.modules[module_name] = module
    self._import_all(module)

    _loaded = False

    def load_ipython_extension(ip):
    """Load the extension in IPython."""
    global _loaded
    if not _loaded:
    ip.register_magics(BiteyMagics)
    _loaded = True