sexta-feira, 30 de dezembro de 2011

Instalando a slackline (sem árvores) com auxílio de âncora - Parte 1

No meu quintal não existem árvores nem outra estrutura para instalar minha slackline.
Pesquisando um pouco descobri o conceito de dead men anchor que nada mais é que uma âncora, no meu caso de madeira, enterrada a 1,5m de profundidade.
Porém, diferentemente desses vídeos, eu não quero enterrar a slackline. Quero uma solução permanente que permita instalar e desinstalar a fita quando quiser.

Por isso resolvi usar cordas de nylon amarradas à âncora. A ideia é amarrar a slackline nessas cordas e usar algum outro tipo de estrutura bem firme: a-frame, cavalete, tronco ou caixa para erguê-la.

Por enquanto, é só um plano. Segue meus esboços:

Assim que começar o projeto posto por aqui!

sábado, 3 de dezembro de 2011

Importing data from Oracle to Google App Engine Datastore with a custom bulkloader connector

I'm working on a project where we need to upload data from Oracle databases to Datastore very frequently.
The datastore bulkloader  bundled in the App Engine Python SDK works very well for data importing and exporting. It currently offers 3 different connector implementations: csv, xml and simpletext.
CSV connector was working smoothly for us but I wanted to avoid the CSV file generation. I wanted to bulkload directly from Oracle.
After some study I decided to write a new connector: the oracle_connector.
The only pre requisite for using this connector is having the cx_Oracle python module properly installed.
Here are some details about my environment:
  • Ubuntu 11.10 - 64 bits
  • Python 2.7.2+
  • cx_Oracle 5.0.4 unicode
  • Oracle Instant Client 11.2
  • Google App Engine Python SDK 1.6.0
Some design decisions

I wanted to specify a sql query for every kind/entity in the yaml config file and then map the returned (selected) column names or aliases to the properties via the external_name attribute. I didn't find a way to use a custom connector_options like "sql_query" so I used the existent columns option to inform the query.
I also wanted to have the database connection properties to be outside the connector implementation. In order to achieve that I decided to store these configurations in an external file which is passed through the command line "--filename" parameter to the appcfg.py script.

The actual implementation

It is basically comprised of 3 files:
  • bulkloader.yaml - imports and utilizes the oracle_connector as well as maps queries to entities
  • oracle_connector.py - connector implementation based on cx_Oracle (doesn't work for exports)
  • db_settings.py - defines connection properties variables used by oracle_connector
Invoking the bulkloader is very simple:

$APPENGINE_HOME/appcfg.py upload_data --config_file=bulkloader.yaml --kind=Table --filename=db_settings.py --email=user@gmail.com --url=http://app-id.appspot.com/remote_api

The bulkloader.yaml below shows how to import and use the oracle_connector. Also note the connector_options attribute.

- import: base64
- import: re
- import: oracle_connector
- import: google.appengine.ext.bulkload.transform
- import: google.appengine.ext.bulkload.bulkloader_wizard
- import: google.appengine.ext.db
- import: google.appengine.api.datastore
- import: google.appengine.api.users


- kind: Table
  connector: oracle_connector.OracleConnector.create_from_options
    columns: "select TABLE_NAME, TABLESPACE_NAME, LAST_ANALYZED from user_tables"
    - property: __key__
      external_name: TABLE_NAME

    - property: tablespace
      external_name: TABLESPACE_NAME

    - property: last_analyzed
      external_name: LAST_ANALYZED

There are some known issues here: 1) you must use uppercase strings in the external_name attribute and 2) I wasn't able to return number columns (had to use to_char oracle function) due to some problem in my cx_Oracle installation.

Below, the OracleConnector class which implements the connector_interface.ConnectorInterface. This was my first piece of python code. Let me know if I can improve it!

#!/usr/bin/env python
"""A bulkloader connector to read data from Oracle selects.
from google.appengine.ext.bulkload import connector_interface
from google.appengine.ext.bulkload import bulkloader_errors
import cx_Oracle
import os.path

class OracleConnector(connector_interface.ConnectorInterface):

  def create_from_options(cls, options, name):
    """Factory using an options dictionary.

      options: Dictionary of options:
        columns: sql query to perform, each selected column becomes a column
      name: The name of this transformer, for use in error messages.

      OracleConnector object described by the specified options.

      InvalidConfiguration: If the config is invalid.
    columns = options.get('columns', None)
    if not columns:
        raise bulkloader_errors.InvalidConfiguration(
            'Sql query must be specified in the columns '
            'configuration option. (In transformer name %s.)' % name)

    return cls(columns)

  def __init__(self, sql_query):

      sql_query: (required) select query which will be sent to database. The returned columns/aliases will be used as the connectors column names
    self.sql_query = unicode(sql_query)

  def generate_import_record(self, filename, bulkload_state):
    """Generator, yields dicts for nodes found as described in the options.

      filename: py script containing oracle database connection properties: host, port, uid, pwd and service.
      bulkload_state: Passed bulkload_state.

      Neutral dict, one per row returned by the sql query
    dbprops = __import__(os.path.splitext(filename)[0])
    dsn_tns = cx_Oracle.makedsn(dbprops.host, dbprops.port, dbprops.service)
    connection = cx_Oracle.connect(dbprops.uid, dbprops.pwd, dsn_tns)
    cursor = connection.cursor()
    cursor.arraysize = dbprops.cursor_arraysize
    num_fields = len(cursor.description)
    field_names = [i[0] for i in cursor.description]
    for row in cursor.fetchall():
       decoded_dict = {}
       for i in range(num_fields):
         decoded_dict[field_names[i]] = row[i]
       yield decoded_dict    

And finally the contents of db_settings.py:

cursor_arraysize = 50

Get the code

You can download the code here:



This post shows an alternative connector implementation for importing data directly from an Oracle database.
This approach could be easily extended to support other RDBMS such as MySQL or Postgres.
We still have to perform some tests to check how it will behave under different loads and data types but so far, it seems promising.


terça-feira, 8 de novembro de 2011

Using google-oauth-java-client to consume a Google App Engine OAuth protected resource

This post shows how to write a java based OAuth client to make requests against a Google App Engine OAuth protected resource using the google-oauth-java-client.

If you want to know how to create an OAuth provider or how to register your domain and get your consumer key and secret, I highly recommend you to read this great blog post written by Ikai Lan.
Actually, you should read it anyway, because the piece of code below just replaces the python script provided by him. Since I needed a java version I decided to write my own client.

The java class below contains a basic junit test which will do the 3-legged OAuth dance. The only thing I haven't implemented is the access token cache part - every time you run this test you will have to explicitly perform the authorization steps in the browser (again).

In  order to run it, you basically have to resolve 2 dependencies:
I used the 1.6.0-beta version. Just download and put them in the classpath and you are good to go!

And, of course, don't forget to change the APP_ID and CONSUMER_SECRET constants.

package com.ciandt.oauth.client;

import java.io.BufferedReader;
import java.io.InputStreamReader;

import org.junit.Test;

import com.google.api.client.auth.oauth.OAuthAuthorizeTemporaryTokenUrl;
import com.google.api.client.auth.oauth.OAuthCredentialsResponse;
import com.google.api.client.auth.oauth.OAuthGetAccessToken;
import com.google.api.client.auth.oauth.OAuthGetTemporaryToken;
import com.google.api.client.auth.oauth.OAuthHmacSigner;
import com.google.api.client.auth.oauth.OAuthParameters;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;

public class OAuthClientTest {

 private static final HttpTransport TRANSPORT = new NetHttpTransport();
 private static final String APP_ID = "your_app_id_here";
 private static final String CONSUMER_KEY = APP_ID + ".appspot.com";
 private static final String CONSUMER_SECRET = "your_consumer_secret_here";
 private static final String PROTECTED_SERVICE_URL = "https://" + APP_ID + ".appspot.com/resource";
 private static final String REQUEST_TOKEN_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthGetRequestToken";
 private static final String AUTHORIZE_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthAuthorizeToken";
 private static final String ACCESS_TOKEN_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthGetAccessToken";

 public void consumeProtectedResource() throws Throwable {

  // this signer will be used to sign all the requests in the "oauth dance"
  OAuthHmacSigner signer = new OAuthHmacSigner();
  signer.clientSharedSecret = CONSUMER_SECRET;

  // Step 1: Get a request token. This is a temporary token that is used for 
  // having the user authorize an access token and to sign the request to obtain 
  // said access token.
  OAuthGetTemporaryToken requestToken = new OAuthGetTemporaryToken(REQUEST_TOKEN_URL);
  requestToken.consumerKey = CONSUMER_KEY;
  requestToken.transport = TRANSPORT;
  requestToken.signer = signer;

  OAuthCredentialsResponse requestTokenResponse = requestToken.execute();
  System.out.println("Request Token:");
  System.out.println("    - oauth_token        = " + requestTokenResponse.token);
  System.out.println("    - oauth_token_secret = " + requestTokenResponse.tokenSecret);

  // updates signer's token shared secret
  signer.tokenSharedSecret = requestTokenResponse.tokenSecret;

  OAuthAuthorizeTemporaryTokenUrl authorizeUrl = new OAuthAuthorizeTemporaryTokenUrl(AUTHORIZE_URL);
  authorizeUrl.temporaryToken = requestTokenResponse.token;
  // After the user has granted access to you, the consumer, the provider will
  // redirect you to whatever URL you have told them to redirect to. You can 
  // usually define this in the oauth_callback argument as well.
  String currentLine = "n";
  System.out.println("Go to the following link in your browser:\n"
    + authorizeUrl.build());
  InputStreamReader converter = new InputStreamReader(System.in);
  BufferedReader in = new BufferedReader(converter);
  while (currentLine.equalsIgnoreCase("n")) {
   System.out.println("Have you authorized me? (y/n)");
   currentLine = in.readLine();
  // Step 3: Once the consumer has redirected the user back to the oauth_callback
  // URL you can request the access token the user has approved. You use the 
  // request token to sign this request. After this is done you throw away the
  // request token and use the access token returned. You should store this 
  // access token somewhere safe, like a database, for future use.
  OAuthGetAccessToken accessToken = new OAuthGetAccessToken(
  accessToken.consumerKey = CONSUMER_KEY;
  accessToken.signer = signer;
  accessToken.transport = TRANSPORT;
  accessToken.temporaryToken = requestTokenResponse.token;

  OAuthCredentialsResponse accessTokenResponse = accessToken.execute();
  System.out.println("Access Token:");
  System.out.println("    - oauth_token        = " + accessTokenResponse.token);
  System.out.println("    - oauth_token_secret = " + accessTokenResponse.tokenSecret);
  System.out.println("\nYou may now access protected resources using the access tokens above.");

  // updates signer's token shared secret
  signer.tokenSharedSecret = accessTokenResponse.tokenSecret;

  OAuthParameters parameters = new OAuthParameters();
  parameters.consumerKey = CONSUMER_KEY;
  parameters.token = accessTokenResponse.token;
  parameters.signer = signer;

  // utilize accessToken to access protected resources
  HttpRequestFactory factory = TRANSPORT.createRequestFactory(parameters);
  GenericUrl url = new GenericUrl(PROTECTED_SERVICE_URL);
  HttpRequest req = factory.buildGetRequest(url);
  HttpResponse resp = req.execute();
  System.out.println("Response Status Code: " + resp.getStatusCode());
  System.out.println("Response body:" + resp.parseAsString());



terça-feira, 11 de outubro de 2011

Automating GAE's application deployment with Jenkins

Follow below the expect script I created to automate the deployment of applications on GAE. It simply invokes/spawns the appcfg.sh and sends the account's password when prompted.
Then I configured a Jenkins job which contains a "execute shell" step invoking this script.

Jenkins "execute shell" step

expect $WORKSPACE/appcfg.exp "$APPENGINE_HOME" "myuser@gmail.com" "mypassword" "update" "$WORKSPACE/war"

appcfg.exp expect script

#!/usr/bin/expect -f
# Expect script to supply GAE's account password for appcfg.sh
# This script needs four arguments:
# username = GAE's google account email
# password = GAE's google account password
# warDir = war directory to deploy to GAE
# gaeHome = GAE's SDK home dir
# For example:
#  expect appcfg.exp myemail@gmail.com mypassword ./war /usr/share/appengine-sdk-1.5.3

if {[llength $argv] == 0} {
   puts "usage: appcfg.exp {-index|#}"
   exit 1

set gaeHome [lrange $argv 0 0]
set username [lrange $argv 1 1]
set password [lrange $argv 2 2]
set cmd [lrange $argv 3 3]
set warDir [lrange $argv 4 4]

set timeout -1

# spawns appcfg.sh
spawn $gaeHome/bin/appcfg.sh --enable_jar_splitting --passin --email=$username $cmd $warDir
match_max 100000

expect {
   default {exit 0}
   # Look for passwod prompt

# Send password aka $password
send -- "$password\r"

# send blank line (\r) to make sure we get back to gui
send -- "\r"
expect eof

Using XmlSlurper to update appengine-web.xml

I'm currently working on a project which is using GAE (Google Application Engine).
I wanted to setup Jenkins' jobs to update the different stages (acceptance, qa, uat) in our build pipeline.
To achieve that I created 3 different GAE's applications - one for each stage - and 4 Jenkins jobs: 1 template which holds the common steps and 3 others for the respective stages.
The first step of the template job updates appengine-web.xml target application and version based on parameters passed to the job.

See the groovy script below:

import groovy.xml.StreamingMarkupBuilder

//getting parameters values from environment variables
def env = System.getenv()
def workspace = env["WORKSPACE"]
def applicationName = env["TARGET_APPLICATION_NAME"]
def versionName = env["TARGET_VERSION_NAME"]
def ant = new AntBuilder()
ant.echo(message:"Opening $workspace/war/WEB-INF/appengine-web.xml")
def file = new File("$workspace/war/WEB-INF/appengine-web.xml")
def root = new XmlSlurper().parse(file)
ant.echo(message:"Updating appengine-web.xml with application: $applicationName and version: $versionName")
def outputBuilder = new StreamingMarkupBuilder()
String result = outputBuilder.bind{ 
   mkp.declareNamespace("":  "http://appengine.google.com/ns/1.0")    
   mkp.yield root 
ant.echo(message:"Writing appengine-web.xml")

quinta-feira, 31 de março de 2011

JMeter - Processing and handling JSON responses using BSF post processors

In one of my Jmeter posts I talked about how to post JSON data using a HttpSampler.
Now I want to show some cool stuff we can do using Jmeter's javascript/BSF support for handling JSON responses.

Suppose we have the following JSON response:

{ success: true, data:[ 1,2,3,4,5 ]  }

This string can be easily converted to a javascript object by using the eval function like this:

eval( 'var myObj = ' + prev.getResponseDataAsString() )

After the evaluation we can use the variable "myObj" as we wish and do things like accessing the success property:


Or iterate over the list in the data property:

for (  var i = 0;  i < myObj.data.length; i++ ) {

We can also put the object in the implicit vars object so it can be used in another sampler, for example:

vars.putObject('myObj', myObj)

sexta-feira, 25 de março de 2011

Generating CSV files using sqlcmd and groovy

Just sharing a simple groovy script I wrote for generating a CSV file from the database.
It depends on SQLServer's sqlcmd command line utility.
It takes 4 input parameters - needed to open a database connection, scans the current directory for .sql files then executes each of them using sqlcmd generating a .temp file. The temp file is then processed - the first and last 2 lines are removed - and renamed to a .csv file.

def username = args[0]
def password = args[1]
def host = args[2]
def database = args[3]
def dir = './'

def ant = new AntBuilder()
def p = ~/.*\.sql/
new File( dir ).eachFileMatch(p) { f ->

    def sqlFile = f.name
    def tempCsvFile = sqlFile.replaceAll(/.sql/,'')

    def cmd = "sqlcmd -S $host -U $username -P $password -d ${database} -i ${sqlFile} -W  -o ${tempCsvFile}.temp -s ;"
    def process = cmd.execute()

    ant.move(file: "${tempCsvFile}.temp", tofile:"${tempCsvFile}.csv", overwrite: true ) {
  headfilter(lines:"-1", skip: "2")
  tailfilter(lines: "-1", skip: "2" )

This could probably be achieved in many other ways.
But it worked like a charm for me!!

terça-feira, 22 de março de 2011

JMeter - POSTing / Sending JSON data

Recently I was load testing a RESTful mobile API where all the transmitted data were encoded as JSON. It took me some time to figure out how to simulate that in JMeter using a regular HTTPRequest sampler so I decided to share it with others.

NOTE: I successfuly used this technique for webservices exposed by the Jersey library as well as ASP.NET asmx webservices

It's quite simple. Assuming you need to send POST request to the /login URI passing two parameters: email and password encoded as JSON, like this:

{ email: 'myemail@email.com', password: 'mypassword' }

Just follow two basic steps:

Step 1 - Setting the "ContentType" http header attribute to "application/json"
  • Add an HTTP Header Manager to your test plan;
  • Add a new attribute: set name to "Content-Type" and value to "application/json"

Step 2 - Setting the JSON object as an unnamed http request parameter
  • Add a HttpRequest sampler
  • Add a new parameter, leave the name blank and set the value to the JSON string:  { 'email': 'myemail@email.com', 'password': 'mypassword' }

And last but not least: you can use variables and properties as you wish in the JSON string. Suppose you have a CSV Data Set Config which defines to external variables: email e pwd. You could use them in your sampler like this:

{ email: '${email}', password: '${pwd}' }