IPython cheat sheet

This is a cheat sheet for IPython 0.11 that I made while watching the excellent Jeff Rush videos on showmedo.com. Use it and enjoy. Here we go:

Part I - basic commands
var - tab completion of a variable that starts with prefix 'var'
who [int, str, ..]- list available variables, or just those with the give type
whos - a table of all the variables, their type and values
psearch var* [type] - search for variables using the pattern and possibly type
reset - clear all variables
logstate - has the logging been started or not
logstart - start logging into (ipython_log.py by default)
logon - turn logging on
logoff - turn logging off
lsmagic - list all built in commands, called magic commands. These are prefixed with % to differentiate between variables if they have the same name.
magic - scrolling help about the magic commands.
cmd? - overview of the command, e.g lsmagic?
cmd?? - more information about the command including source code
var? - information about the variable
page obj - pretty print the object
func arg1, arg2 - automatically adds parentheses and calls func(arg1, arg2)
/func - same as func()
, func arg1, arg2 - same as func("arg1", "arg2")
help(keyword) - manual pages on keyword (module, function, etc)
help() -> topics - lists all the help topics
pdef func - shows the function signature (list of arguments it takes)
pdoc func - prints doc string for the function
pinfo func - same as func?
psource func - shows the source code for the function
pfile func - opens the source file with the func
edit -x func - opens the source with default editor on the line with the function

Part  II - navigation and the shell
Analogous to standard shell commands there are pwd, cd, pushd, popd, dirs Tab completion also works with these.
bookmark name [path] - bookmarks the current directory or the given path
bookmark -l - lists all saved bookmarks
bookmark -d name - removes bookmark name
dhist - prints the directory navigation history. cd can take numbers from that list as argument, e.g cd -2
!cmd - runs cmd as a shell command, e.g !ls
alias - lists defined aliases for shell commands we can use
alias newname cmd args - creates a new alias with newname as name for the cmd with args
unalias name - removes alias name
var = !cmd - store the output of cmd into variable
pycat name - prints the highlighted contents of the file
run script.py - runs a python file called script.py and prints the output. Also the functions, variables and modules from the script REMAIN in our namespace!
run -p script.py - run the profiling on the script

Part  III - input manipulation
ctrl+p, ctrl+n, arrow keys - previous next command from history
_i, _ii, _iii - prints 1,2 or 3 previous command
_i[X] - prints command number X, e.g _i1
In[X] - prints X-th input commands
hist [N1-N2] - prints a list of previously typed commands, or N1 through N2 previous commands
macro name cmdA-cmdB cmdX - defines a mactor that can execute commands from the history
print macro - prints a macro
edit file - opens up the default editor to edit a file
edit [N1-N2] - edit the history N1 through N2
save file N1-N2 - saves the historey N1 through N2 into file.py
cpaste - paste in source code without, IPython will not format it.
Out[X] - prints the X-th output if it exists
func; - putting does not echo the return of the function and does not put it into Out dictionary



Note! The following commands that Jeff uses in version 0.8.1 do not work in IPython 0.11:
store var - save the variable in profile for future use
p var - shorthand for print
exec cmd - execute a command
runlog - executes the IPython log

Accepted to GSoC!

Awesome! My summer of code application got accepted by SIP Communicator! It's going to be a hot summer!
In fact, the dev team decided that using environment specific password storage is not a such a good idea after all, and it would be much better to do something more generic. I agree, and in fact Java provides with the means to do so.
Since implementing only the generic password storage is clearly not enough for the whole summer, I'll also be doing another project, call recording. I really don't know much how to go about doing this right now, but I'm sure I'll think of something and it's always nice to learn something new like Java Media Framework API.
Right now however I'll concentrate on the password storage and try to accomplish that as soon as possible. First thing first, I should make and overview of the new requirements. Stay tuned.

SIP Communicator GSoC 2010 Application Part 5

The Result

It's time to think about how to put everything together. Clearly there must be some generic solution that allows easily adding new external password storage utilities such as KWallet or OSX utility. Well, the obvious way to do this is to use something like the following from the ProtocolProviderFactory.storePassword (which I talked about in Part 2). Just a sketch:

 public static class PasswordStore {
  public void storePassword(String account, String password) {
   Runtime runtime = Runtime.getRuntime();
   if (OSUtils.IS_LINUX) {
    // can we check if gnome is running in some other way?
    if (runtime.exec(new String[] { "which", "gnome-keyring" })
      .waitFor() == 0) {
     password = new GnomeKeyring().storePassword(account,
       password);
     // probably add a property that says we're using
     // gnome-keyring,
     // this would make loading easier
    } else if (runtime.exec(new String[] { "which", "kwallet" })
      .waitFor() == 0) {
     // store to KWallet
    }
   } else if (OSUtils.IS_MAC) {
    // osx storage
   } else {
    // fall back to default behaviour (base-64 encode)
   }
  }
 }

I did a quick search in the code and it seems that OSUtils.IS_* is used everywhere to check the OS type. I didn't find anything to check what desktop environment is used, but inside BrowserLauncherImpl I found similar code that finds browsers using unix which command, that's one way I guess, but maybe there's a better solution.
Anyway, the idea is that when the password is stored a new property is written in the configuration that tells what storage mechanism is used. With this property present, password loading is easy. Sketch:

 public static class PasswordLoad {
  public String loadPassword(String account) {
   String password = null;
   String property = configurationService.getString(storageType);
   if (property.equals("gnome-keyring")) {
    password = new GnomeKeyRing().loadPassword(account);
   } else {
    // fall back to default and get password from the properties
   }
   return password;
  }
 }

Clearly, default behavior should be used when no other option is possible. Also, maybe when the password was not found in gnome-keyring we should also look in the configuration, just in case.

Concerning KWallet, I haven't researched it much (being a gnome user), but there is this thing. A whole java library for KWallet, excellent! So, I guess it should not be a problem to talk to it. I certainly can do that if gnome-keyring is already working and integrated.

Well, that seems to be it. I've described in 5 parts what I would do to implement a secure password storage with GnomeKeyring for SIP Communicator. I think the application is complete and I really need some feedback from the development team. If they like it and my application becomes accepted I'll continue posting on my progress here.

SIP Communicator GSoC 2010 Application Part 4

The Progress

I managed to get JNI to talk to GnomeKeyring via libgnome-keyring. Basically I can store, delete, load passwords, check GK availability and list it's keyrings. Here's the code:
First the java part:

package example;


public class GnomeKeyring {
 public native void setApplicationName(String name);
 public native String[] getKeyRingNames();
 public native boolean isKeyRingAvailable();
 public native boolean storePassword(String account, String pass);
 public native boolean deletePassword(String account);
 public native String loadPassword(String account);

 static {
     System.loadLibrary("gk-native");
 }

 public GnomeKeyring(String appName) {
     setApplicationName(appName);
 }

 public static void main(String[] args) {
     String libPath = System.getProperty("java.library.path");
     System.out.println("java.library.path = " + libPath);

     GnomeKeyring gk = new GnomeKeyring("SIP Communicator");
 
     System.out.println("Keyring available? " + gk.isKeyRingAvailable());
     System.out.println("Keyrings: ");
     for (String s : gk.getKeyRingNames()) {
         System.out.println(s);
     }
     if (gk.storePassword("dmitri@msn.com", "secret")) {
         System.out.println("Password stored OK");
     }
     System.out.println("Pass found: " + gk.loadPassword("dmitri@msn.com"));
     if (gk.deletePassword("dmitri@msn.com")) {
         System.out.println("Password deleted OK");
     }
 }

}

Pretty simple, it loads the custom shared library gk-native and calls native methods.

The C part is bigger:

#include <jni.h>
#include <stdio.h>
#include <gnome-keyring.h>
#include <glib.h>
#include <libintl.h>
#include "GnomeKeyring.h"

JNIEXPORT jstring JNICALL Java_example_GnomeKeyring_loadPassword
(JNIEnv *env, jobject obj, jstring jaccount) {
    gchar *password;
    jstring jpass = NULL;
    const char *account = (*env)->GetStringUTFChars(env, jaccount, 0);
    GnomeKeyringResult result = gnome_keyring_find_password_sync(GNOME_KEYRING_NETWORK_PASSWORD,
                                                                 &password,
                                                                 "user",     account,
                                                                 /* Can put other properties here that
                                                                  * GNOME_KEYRING_NETWORK_PASSWORD schema supports.
                                                                  */
                                                                 NULL
                                                                 );
    (*env)->ReleaseStringUTFChars(env, jaccount, account);
    
    if (result == GNOME_KEYRING_RESULT_OK) {
        jpass = (*env)->NewStringUTF(env, password);
    }

    gnome_keyring_free_password(password);
    return jpass;
}

JNIEXPORT jboolean JNICALL Java_example_GnomeKeyring_deletePassword
(JNIEnv *env, jobject obj, jstring jaccount) {
    const char *account = (*env)->GetStringUTFChars(env, jaccount, 0);
    GnomeKeyringResult result = gnome_keyring_delete_password_sync(GNOME_KEYRING_NETWORK_PASSWORD,
                                                                   "user",     account,
                                                                   /* Can put other properties here that
                                                                    * GNOME_KEYRING_NETWORK_PASSWORD schema supports.
                                                                    */
                                                                   NULL
            );
    (*env)->ReleaseStringUTFChars(env, jaccount, account);

    return result == GNOME_KEYRING_RESULT_OK ? TRUE : FALSE;
}

JNIEXPORT jboolean JNICALL Java_example_GnomeKeyring_storePassword
(JNIEnv *env, jobject obj, jstring jaccount, jstring jpass) {
    const char *account = (*env)->GetStringUTFChars(env, jaccount, 0);
    const char *pass = (*env)->GetStringUTFChars(env, jpass, 0);
    GnomeKeyringResult result = gnome_keyring_store_password_sync(GNOME_KEYRING_NETWORK_PASSWORD,
                                                                  NULL,
                                                                  "My Password",
                                                                  pass,
                                                                  "user",     account,
                                                                  /* Can put other properties here that
                                                                   * GNOME_KEYRING_NETWORK_PASSWORD schema supports.
                                                                   */
                                                                  NULL
            );
    (*env)->ReleaseStringUTFChars(env, jaccount, account);
    (*env)->ReleaseStringUTFChars(env, jpass, pass);

    return result == GNOME_KEYRING_RESULT_OK ? TRUE : FALSE;
}

JNIEXPORT void JNICALL Java_example_GnomeKeyring_setApplicationName
(JNIEnv *env, jobject obj, jstring name) {
    const char *str = (*env)->GetStringUTFChars(env, name, 0);
    g_set_application_name(str);
    (*env)->ReleaseStringUTFChars(env, name, str);
}

JNIEXPORT jobjectArray JNICALL Java_example_GnomeKeyring_getKeyRingNames
(JNIEnv *env, jobject obj)
{
    int i;
    char *keyring;
    GList *keyrings, *l;
    jstring string = NULL;

    gnome_keyring_list_keyring_names_sync (&keyrings);
    guint len = g_list_length(keyrings);

    jclass stringClass = (*env)->FindClass(env,"Ljava/lang/String;");
    jobjectArray stringArray = (*env)->NewObjectArray(env, len, stringClass, NULL);
    for (l = keyrings, i = 0; l != NULL; l = l->next, i++) {
        keyring = l->data;
        string = (*env)->NewStringUTF(env, keyring);
        (*env)->SetObjectArrayElement(env, stringArray, i, string);
        (*env)->DeleteLocalRef(env, string);
    }
    gnome_keyring_string_list_free(keyrings);

    return stringArray;
}

JNIEXPORT jboolean JNICALL Java_example_GnomeKeyring_isKeyRingAvailable
(JNIEnv *env, jobject obj)
{
    if(gnome_keyring_is_available()) {
        return TRUE;
    }
    return FALSE;
}

Here I used only "user" property of the GNOME_KEYRING_NETWORK_PASSWORD schema, but other can be used as well, or even new schemas can be created.
Finally the makefile to put it all together:

BIN=bin
PAK=example
CSRC=src/$(PAK)/native

JFILE=GnomeKeyring
CFILE=$(JFILE).c
HFILE=$(JFILE).h
OBJ=$(JFILE).o

LIB=libgk-native.so

all : $(BIN)/$(LIB)

$(BIN)/$(LIB) : $(BIN)/$(OBJ)
 gcc -shared -o $(BIN)/$(LIB) $(BIN)/$(OBJ) /usr/lib/libgnome-keyring.so 

$(BIN)/$(OBJ) : $(CSRC)/$(CFILE) 
 gcc -I/usr/lib/jvm/java-6-sun/include -I/usr/lib/jvm/java-6-sun/include/linux -I/usr/include/gnome-keyring-1 -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -c -fpic $(CSRC)/$(CFILE) -o $(BIN)/$(OBJ)

header : $(BIN)/$(PAK)/$(JFILE).class
 javah -jni -classpath bin -o $(CSRC)/$(HFILE) $(PAK).$(JFILE) 

run:
 java -cp $(BIN) -Djava.library.path=$(BIN) $(PAK).$(JFILE)
clean :
 rm $(BIN)/$(OBJ)
 rm $(BIN)/$(LIB)

So now I can do everything that's required with GnomeKeyring, next I will try to think of a way to integrate it all into SC.

SIP Communicator GSoC 2010 Application Part 3

The JNI Way

First of all I started researching Seahorse and it's DBUS API. I was hoping that it's possible to use the java bindings for dbus to store and retrieve passwords but then I realized that in fact it was only possible to store keys with Seahorse dbus and not passwords, so the java dbus api was of no use. I wasted a lot of time on all of this.
My only option was to use libgnome-keyring. Because it has only C and python bindings, I had to use it via JNI. Not knowing anything about Java Native Interface, I started researching again and after a lot of googling and experimenting finally came up with a custom shared library (my C file with a function that lists all keyrings) linked together with libgnome-keyring, and a small java class that loads it and calls the function. It took time and patience but it works - java can talk to GnomeKeyring! Next I'll try password store and retrieve.

SIP Communicator GSoC 2010 Application Part 2

The Problem
This part describes what the project idea is all about and what needs to be done.
The thing with security is that a system is secure only as much as it's weakest link. That weakest link happens to be the password most of the time. Not only can it be easy to guess, even worst, it can be SEEN by simply knowing a thing or two about how the system works. SIP Communicator (SC) stores it's passwords insecurely in a Base64-encoded string inside ~/.sip-communicator/sip-communicator.properties. Just grep for PASSWORD=, run base64 -d on that value and you're done, you have the password for the account. Sure, only the user can read files in his home directory. But insecure still and good software needs good security.
If we go deep into the code then here's what we have (mostly taken from my letter to SC dev list):

AccountManager loads the accounts on start-up in a separate thread. It's doLoadStoredAccounts() method is called for all supported
ProtocolProviderFactory implementations, and this method (besides other things) decodes the stored password and puts it in a map along with other properties; the map is then passed into the ProtocolProviderFactory.loadAccount() is done for all accounts for the given protocol. Now, inside ProtocolProviderFactory.loadAccount, a protocol specific AccountID is created from the passed-in properties map.
After that, things get interesting: in the same method ServiceRegistration object is created with the call bundleContext.registerService(...). This triggers inside a new thread a call chain from ProtocolProviderFactory*Impl.register() (which has our created AccountID with the password inside) to ProtocolProviderFactory.loadPassword(bundleContext, accountID) which does the same thing as doLoadStoredAccounts() - loads and decodes the password.

Actually, it seems that doLoadStoredAccounts() password decoding is some old code that should be deleted, I commented it out and it ran ok. So the things that needs to be refactored to support third party password storage utilities is ProtocolProviderFactory's loadPassword and storePassword methods.

Of course we still must preserve the current plaintext storage because not everyone runs GnomeKeyring/Kwallet/other utility, but we still must have a system that can identify what password storage utility is available and use it or fall back to the default plaintext base64 storage.

SIP Communicator GSoC 2010 Application Part 1

The Overview

This year I decided to apply for Google Summer of Code and the organization I chose is SIP Communicator. It's a really good project with a great development team that is basically an universal IM that supports everything from MSN to IRC and Facebook. It's written in Java and uses Apache Felix OSGi implementation.
The project idea I want to work on is GnomeKeyring (since I use Gnome) for password storage. I've always been interested in security and this semester I'm taking a really good course on applied cryptography using Java, so it's a perfect thing for me.
All of my application related things will be in this blog because I find it more convenient to just give a link that points here and keep everything in one place. If I get accepted, then I will continue to post on my progress here.
In the next posts, I'm going to present the technical details of my application, stay tuned.