How to accurately test code in Python?

Question

I am trying to compare some custom C modules with the standard Python modules. For instance, I wanted to test if a random function in C could out-perform Pythons randint method.
I have tried using the timeit module, but I am curious if I am skewing my own results in any way.
import timeit
import ctypes
import random
my_test_lib = ctypes.cdll.LoadLibrary('/home/name/workspace/Testing/test.so')
crand_timer = timeit.Timer("crand()", "from __main__ import crand")
prand_timer = timeit.Timer("prand()", "from __main__ import prand")
def crand():
for i in range(10):
my_test_lib.get_random(1, 10)
def prand():
for i in range(10):
random.randint(1,10)
if __name__ == '__main__':
print crand_timer.timeit()
print prand_timer.timeit()
Just for kicks, here is the C code:
#include <stdlib.h>
#include <time.h>
#define TRUE 1
#define FALSE 0
static int randomized = FALSE;
void randomize(void)
srand(time(NULL));
randomized = TRUE;
/* returns a random number between base and max, inclusive */
int get_random(int base, int max)
if (randomized == FALSE){
randomize();
return rand() % (max - base + 1) + base;
I compiled the C code with O3. The results seem to indicated the C code is about 2.3 times faster than the Python randint method.
Am I skewing the results in any way? I have a large amount of C code to test. I will be integrating a bunch of low level C modules into a Python program I am going to build. I want to learn how to test efficiently before I start.
edit:
If anyone needs/wants/interested in knowing-- I plan to build a tile based game with multi-threaded procedural dungeon generator. I want to build the generator in C and make it as fast as possible.
Last edited by Google (2011-10-27 07:21:54)

Answer 1

Hi,
If you increase the samples (say loop 5000 times instead of 10), the answers will be better, but accurate benchmarking is really hard.
Have you seen The Computer Language Benchmarks Game. I find it interesting.
When it comes to compilation flags, -O2 vs -O3 is seldom the problem. It's hard to choose the best algorithm and it's hard to make it scale as intented.
Do you know of Big O notation? That could be helpful.
Good luck with the game and dungeon generator.