Páginas

SyntaxHighlighter

Mostrando postagens com marcador java. Mostrar todas as postagens
Mostrando postagens com marcador java. Mostrar todas as postagens

sábado, 27 de julho de 2013

Dynamically generating Zip files using Google Cloud Storage Client Library for Appengine

Let's say you have 50 objects (15Mb each) stored in Google Cloud Storage. Now, you need to create a zip archive containing all of them and store the resultant file back into GCS. How can we achieve that from within an Appengine java application?
Well, after some research, I wrote the method below using Google Cloud Storage Client Library which does exactly that. Just don't forget  to grant the appropriate permissions to your appengine service account so that it can read and write the objects.


public static void zipFiles(final GcsFilename targetZipFile,
  final GcsFilename... filesToZip) throws IOException {

 Preconditions.checkArgument(targetZipFile != null);
 Preconditions.checkArgument(filesToZip != null);
 Preconditions.checkArgument(filesToZip.length > 0);

 final int fetchSize = 4 * 1024 * 1024;
 final int readSize = 2 * 1024 * 1024;
 GcsOutputChannel outputChannel = null;
 ZipOutputStream zip = null;
 try {
  final GcsFileOptions options = new GcsFileOptions.Builder()
    .mimeType(MediaType.ZIP.toString()).build();
  outputChannel = GCS_SERVICE.createOrReplace(targetZipFile, options);
  zip = new ZipOutputStream(Channels.newOutputStream(outputChannel));
  GcsInputChannel readChannel = null;
  for (final GcsFilename file : filesToZip) {
   try {
    final GcsFileMetadata meta = GCS_SERVICE.getMetadata(file);
    if (meta == null) {
     LOGGER.warning(file.toString()
       + " NOT FOUND. Skipping.");
     continue;
    }
    final ZipEntry entry = new ZipEntry(file.getObjectName());
    zip.putNextEntry(entry);
    readChannel = GCS_SERVICE.openPrefetchingReadChannel(file,
      0, fetchSize);
    final ByteBuffer buffer = ByteBuffer.allocate(readSize);
    int bytesRead = 0;
    while (bytesRead >= 0) {
     bytesRead = readChannel.read(buffer);
     buffer.flip();
     zip.write(buffer.array(), buffer.position(),
       buffer.limit());
     buffer.rewind();
     buffer.limit(buffer.capacity());
    }

   } finally {
    zip.closeEntry();
    readChannel.close();
   }
  }
 } finally {
  zip.flush();
  zip.close();
  outputChannel.close();
 }
}

sábado, 6 de julho de 2013

"Smart" Appengine Devserver restarts for faster development lifecycle

We just started a new AppEngine Java project using the appengine-maven-plugin and the Eclipse IDE (Juno). We are using the appengine:devserver goal to start the devserver. It basically builds the entire project (compile, test and package) and after that launches the devserver pointing it to the generated webapp directory - which by default is: ${project.build.directory}/${project.build.finalName}


The Problem

Every edit made in the source directory is not recognized unless the server is restarted - neither static content nor newly compiled classes. Which is obvious since the devserver is monitoring only the target directory.
It's a very "bureaucratic" and nonproductive development environment - we need to stop and start the server even for a single CSS line change.


The Dream

Achieve the same productivity level we have when working with dynamic languages based development environment (i.e. Python). Just hit F5 in the browser to see the changes in static files and automatic server reload every time a Java class or descriptor file is compiled/changed.


The Solution 

Using a little Ant-foo, we were able to create a target which synchronizes both directories.
Thie snippet below uses the sync Ant task to perform the static content synchronization (lines 1-9). Notice that everything inside src.webapp.dir is sync'ed except for the 3 directories declared in the preserveintarget element. We had to exclude them from the synchronization process because since they only exist in the target directory they'd be deleted otherwise. And finally a second sync to synchronize the compiled classes (lines 11-13).

<sync verbose="true" todir="${target.webapp.dir}" includeEmptyDirs="true">
 <fileset dir="${src.webapp.dir}" />
 <preserveintarget>
     <!-- Ignore the directories below -->
  <include name="WEB-INF/lib/**" />
  <include name="WEB-INF/classes/**" />
  <include name="WEB-INF/appengine-generated/**" />
 </preserveintarget>
</sync>

<sync verbose="true" todir="${target.webapp.dir}/WEB-INF/classes">
 <fileset dir="${basedir}/target/classes" />
</sync>

Then, we attached it to an Eclipse builder, which is triggered every time a change is made in the project ("Build automatically" flag enabled).
The same behavior can be achieved by creating a special maven profile and using a combination of the m2e lifecycle mappings and the maven antrun plugin.
Something like this:

<profile>
    <id>m2e</id>
    <activation>
        <property>
            <name>m2e.version</name>
        </property>
    </activation>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-antrun-plugin</artifactId>
                <version>1.7</version>
                <executions>
                    <execution>
                        <phase>process-classes</phase>
                        <goals>
                            <goal>run</goal>
                        </goals>
                        <configuration>
                            <target>
                                <property name="target.webapp.dir" value="${project.build.directory}/${project.build.finalName}" />
                                <property name="src.webapp.dir" value="${basedir}/src/main/webapp" />
                                <sync verbose="true" todir="${target.webapp.dir}" includeEmptyDirs="true">
                                    <fileset dir="${src.webapp.dir}" />
                                    <preserveintarget>
                                        <include name="WEB-INF/lib/**" />
                                        <include name="WEB-INF/classes/**" />
                                        <include name="WEB-INF/appengine-generated/**" />
                                    </preserveintarget>
                                </sync>
                                <sync verbose="true" todir="${target.webapp.dir}/WEB-INF/classes">
                                    <fileset dir="${basedir}/target/classes" />
                                </sync>
                            </target>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <!-- This plugin's configuration is used to store Eclipse m2e settings 
      only. It has no influence on the Maven build itself. -->
                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>org.apache.maven.plugins</groupId>
                                        <artifactId>maven-antrun-plugin</artifactId>
                                        <versionRange>[1.6,)</versionRange>
                                        <goals>
                                            <goal>run</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <execute>
                                            <runOnIncremental>true</runOnIncremental>
                                        </execute>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</profile>

sexta-feira, 6 de julho de 2012

API Java para busca de CEPs (fachada para o serviços dos Correios)

O site dos Correios disponibiliza gratuitamente um ótimo serviço para busca online de CEPs.
Porém, essa é a única forma de acessá-lo: via página HTML.
Não existe, nem é exposta, nenhuma API que facilite a integração desse serviço com outras aplicações.

Para possibilitar esse tipo de integração, criei o busca-cep-java-client que nada mais é que um componente Java (jar) cuja API abstrai a complexidade de:
  1. Fazer a requisição HTTP (GET) passando os parâmetros necessários e;
  2. Processar a resposta - extraindo os dados de CEP do HTML retornado. 
Internamente utilizo a biblioteca HtmlUnit para auxiliar no Web Scraping. Basicamente o que eu faço é simular um usuário que entra na página de busca, preenche e posta o form. Depois, já na tela de resultados, percorro o HTML retornado em busca da tabela com os dados de CEP retornados.

O trecho de código abaixo mostra o quão simples é a utilização da API:

// Obtém uma instância de CEPService
CEPService buscaCEP = CEPServiceFactory.getCEPService();
// Obtém um CEP pelo número
CEP cep = buscaCEP.obtemPorNumeroCEP(13084440);
// Obtém todos os CEPs que contém "Flordalisa" no logradouro
List<CEP> ceps = buscaCEP.obtemPorEndereco("Flordalisa");

Para mais detalhes em como utilizar esse componente, sugiro dar uma olhada nos testes unitários.
Todo o código fonte também está disponível no Github.
Além disso, publiquei a versão 1.1 do componente num repositório Maven (também no Github).
Para quem usa essa ferramenta basta alterar o pom.xml para incluir o repositório:

<repository>
        <id>Talesolutions</id>
        <url>https://raw.github.com/fabito/talesolutions-mvn-repo/master/</url>
        <snapshots>
                <enabled>true</enabled>
        </snapshots>
        <releases>
                <enabled>true</enabled>
        </releases>
</repository>

E adicionar essa dependência:

    <dependency>
      <groupId>org.talesolutions</groupId>
      <artifactId>busca-cep-client</artifactId>
      <version>1.1</version>
    </dependency>

Considerações Finais

O busca-cep-client ajuda a integrar funcionalidades de busca de CEPs à aplicações Java. Principalmente porque dispensa a criação de uma base de CEPs local e como é só uma façade para o serviço oficial dos Correios temos a "garantia" de estarmos sempre acessando dados atuais e corretos.
A principal desvantagem é a dependência da disponibilidade e formato desse serviço - alto acoplamento. Se o site dos Correios sair do ar sua aplicação certamente será impactada. Além disso, como os dados de CEPs são extraidos de uma página HTML, qualquer mudança no markup gerado afetará o funcionamento do componente.

terça-feira, 8 de novembro de 2011

Using google-oauth-java-client to consume a Google App Engine OAuth protected resource

This post shows how to write a java based OAuth client to make requests against a Google App Engine OAuth protected resource using the google-oauth-java-client.

If you want to know how to create an OAuth provider or how to register your domain and get your consumer key and secret, I highly recommend you to read this great blog post written by Ikai Lan.
Actually, you should read it anyway, because the piece of code below just replaces the python script provided by him. Since I needed a java version I decided to write my own client.

The java class below contains a basic junit test which will do the 3-legged OAuth dance. The only thing I haven't implemented is the access token cache part - every time you run this test you will have to explicitly perform the authorization steps in the browser (again).

In  order to run it, you basically have to resolve 2 dependencies:
I used the 1.6.0-beta version. Just download and put them in the classpath and you are good to go!

And, of course, don't forget to change the APP_ID and CONSUMER_SECRET constants.

package com.ciandt.oauth.client;

import java.io.BufferedReader;
import java.io.InputStreamReader;

import org.junit.Test;

import com.google.api.client.auth.oauth.OAuthAuthorizeTemporaryTokenUrl;
import com.google.api.client.auth.oauth.OAuthCredentialsResponse;
import com.google.api.client.auth.oauth.OAuthGetAccessToken;
import com.google.api.client.auth.oauth.OAuthGetTemporaryToken;
import com.google.api.client.auth.oauth.OAuthHmacSigner;
import com.google.api.client.auth.oauth.OAuthParameters;
import com.google.api.client.http.GenericUrl;
import com.google.api.client.http.HttpRequest;
import com.google.api.client.http.HttpRequestFactory;
import com.google.api.client.http.HttpResponse;
import com.google.api.client.http.HttpTransport;
import com.google.api.client.http.javanet.NetHttpTransport;

public class OAuthClientTest {

 private static final HttpTransport TRANSPORT = new NetHttpTransport();
 
 private static final String APP_ID = "your_app_id_here";
 private static final String CONSUMER_KEY = APP_ID + ".appspot.com";
 private static final String CONSUMER_SECRET = "your_consumer_secret_here";
 
 private static final String PROTECTED_SERVICE_URL = "https://" + APP_ID + ".appspot.com/resource";
 private static final String REQUEST_TOKEN_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthGetRequestToken";
 private static final String AUTHORIZE_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthAuthorizeToken";
 private static final String ACCESS_TOKEN_URL = "https://" + APP_ID + ".appspot.com/_ah/OAuthGetAccessToken";

 @Test
 public void consumeProtectedResource() throws Throwable {

  // this signer will be used to sign all the requests in the "oauth dance"
  OAuthHmacSigner signer = new OAuthHmacSigner();
  signer.clientSharedSecret = CONSUMER_SECRET;

  // Step 1: Get a request token. This is a temporary token that is used for 
  // having the user authorize an access token and to sign the request to obtain 
  // said access token.
  OAuthGetTemporaryToken requestToken = new OAuthGetTemporaryToken(REQUEST_TOKEN_URL);
  requestToken.consumerKey = CONSUMER_KEY;
  requestToken.transport = TRANSPORT;
  requestToken.signer = signer;

  OAuthCredentialsResponse requestTokenResponse = requestToken.execute();
  
  System.out.println("Request Token:");
  System.out.println("    - oauth_token        = " + requestTokenResponse.token);
  System.out.println("    - oauth_token_secret = " + requestTokenResponse.tokenSecret);

  // updates signer's token shared secret
  signer.tokenSharedSecret = requestTokenResponse.tokenSecret;

  OAuthAuthorizeTemporaryTokenUrl authorizeUrl = new OAuthAuthorizeTemporaryTokenUrl(AUTHORIZE_URL);
  authorizeUrl.temporaryToken = requestTokenResponse.token;
  
  // After the user has granted access to you, the consumer, the provider will
  // redirect you to whatever URL you have told them to redirect to. You can 
  // usually define this in the oauth_callback argument as well.
  String currentLine = "n";
  System.out.println("Go to the following link in your browser:\n"
    + authorizeUrl.build());
  InputStreamReader converter = new InputStreamReader(System.in);
  BufferedReader in = new BufferedReader(converter);
  while (currentLine.equalsIgnoreCase("n")) {
   System.out.println("Have you authorized me? (y/n)");
   currentLine = in.readLine();
  }
  
  // Step 3: Once the consumer has redirected the user back to the oauth_callback
  // URL you can request the access token the user has approved. You use the 
  // request token to sign this request. After this is done you throw away the
  // request token and use the access token returned. You should store this 
  // access token somewhere safe, like a database, for future use.
  OAuthGetAccessToken accessToken = new OAuthGetAccessToken(
    ACCESS_TOKEN_URL);
  accessToken.consumerKey = CONSUMER_KEY;
  accessToken.signer = signer;
  accessToken.transport = TRANSPORT;
  accessToken.temporaryToken = requestTokenResponse.token;

  OAuthCredentialsResponse accessTokenResponse = accessToken.execute();
  System.out.println("Access Token:");
  System.out.println("    - oauth_token        = " + accessTokenResponse.token);
  System.out.println("    - oauth_token_secret = " + accessTokenResponse.tokenSecret);
  System.out.println("\nYou may now access protected resources using the access tokens above.");

  // updates signer's token shared secret
  signer.tokenSharedSecret = accessTokenResponse.tokenSecret;

  OAuthParameters parameters = new OAuthParameters();
  parameters.consumerKey = CONSUMER_KEY;
  parameters.token = accessTokenResponse.token;
  parameters.signer = signer;

  // utilize accessToken to access protected resources
  HttpRequestFactory factory = TRANSPORT.createRequestFactory(parameters);
  GenericUrl url = new GenericUrl(PROTECTED_SERVICE_URL);
  HttpRequest req = factory.buildGetRequest(url);
  HttpResponse resp = req.execute();
  System.out.println("Response Status Code: " + resp.getStatusCode());
  System.out.println("Response body:" + resp.parseAsString());

 }

}