JoeJag :: Tech

Feb

How to send a raw HTTP request via Java

by Joe Wright

While trying to figure out how a service worked I recently had to put together a Java class that let you easily replay a http conversation which had been sniffed.

I got bored of using telnet while making small changes to the payload and curl wants requests converted into an XML format it appears.

This simple socket based class lets you send a captured http request to a service. I’ve removed the exception handling for brevity.

import java.io.*;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;

public class Client {

    public static void main(String[] args) throws IOException {
        Socket socket = new Socket(args[0], 80);

        BufferedWriter out = new BufferedWriter(
                new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
        BufferedReader in = new BufferedReader(
                new InputStreamReader(socket.getInputStream()));

        sendMessage(out, new File(args[1]));
        readResponse(in);

        out.close();
        in.close();
    }

    private static void sendMessage(BufferedWriter out, File request) throws IOException {
        System.out.println(" * Request");

        for (String line : getContents(request)) {
            System.out.println(line);
            out.write(line + "\r\n");
        }

        out.write("\r\n");
        out.flush();
    }

    private static void readResponse(BufferedReader in) throws IOException {
        System.out.println("\n * Response");

        String line;
        while ((line = in.readLine()) != null) {
            System.out.println(line);
        }
    }

    private static List getContents(File file) throws IOException {
        List contents = new ArrayList();

        BufferedReader input = new BufferedReader(new FileReader(file));
        String line;
        while ((line = input.readLine()) != null) {
            contents.add(line);
        }
        input.close();

        return contents;
    }
}

For example. Running this class with the parameters: “google.co.uk /path/to/stored/file” with the stored file being just:

GET /intl/en/policies/privacy/ HTTP/1.1

Will give you this output:

 * Sending
GET /intl/en/policies/privacy/ HTTP/1.1

 * Response
HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: text/html
Last-Modified: Fri, 27 Jan 2012 17:53:03 GMT
Date: Tue, 07 Feb 2012 21:40:30 GMT
Expires: Tue, 07 Feb 2012 21:40:30 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
Server: sffe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked

// Body content of web page

For capturing payloads I recommend using tcpflow running on the target host as “tcpflow port 80″

Nov

Checking JMX with JRuby

by Joe Wright

I’ve recently been playing around with JRuby and found the switch from MRI to be hassle free. One of the advantages of JRuby is it’s portability. You can use nearly all Ruby code straight away on the JVM without having to do any code changes, you just include the jruby jar in your classpath.

Another advantage is Java interoperability from JRuby (just require ‘java’). I’ve found this useful when using JMX to check certain conditions are in place while performing releases and for general monitoring.

The following script uses JRuby’s Java integration to walk a given ObjectName’s attributes within JMX.

require 'java'

# Return a JMX Connection
def connect_to_jmx service_url, credentials
	jmxUrl = javax.management.remote.JMXServiceURL.new(service_url)

	environment = java.util.HashMap.new
	environment.put(javax.management.remote.JMXConnector.CREDENTIALS,
                              credentials.to_java(:string))

	jmxCon = javax.management.remote.JMXConnectorFactory.connect(jmxUrl,
                                                                                              environment)
	return jmxCon.getMBeanServerConnection
end

# Walk the JMX tree and return the attributes.  Optionally limit to a single ObjectName
def walk_tree jmx_connection, limited_to=''
	names = jmx_connection.queryNames(
                                           javax.management.ObjectName.new(limited_to), nil)

	cached = Hash.new([])

	names.each do |name|
	   info = jmx_connection.getMBeanInfo(name)
	   info.getAttributes.each do | mbi|
		   attr = jmx_connection.getAttribute(name, mbi.getName)
		   cached.store(name.getCanonicalName,
                          cached[name.getCanonicalName] << { mbi.getName  => attr })
	   end
	end

	return cached
end

service_url = 'host:port'
credentials = ['username', 'password']
jmx_connection = connect_to_jmx(service_url, credentials)

results = walk_tree(jmx_connection, 'java.lang:type=OperatingSystem')

results.each do | key, attributes |
  puts ' * ' + key
  attributes.each do | attr |
	puts " ** #{attr.first[0]} :: #{attr.first[1]}"
  end
  puts
end

When invoked against a running JMX instance this will return something similar to:

> jruby jmx_walker.rb
 * java.lang:type=OperatingSystem
 ** FreePhysicalMemorySize :: 51955511296
 ** TotalPhysicalMemorySize :: 151873998848
 ** Name :: Linux
 ** Arch :: amd64
 ** Version :: 2.6.18-194.11.4.el5
 ** AvailableProcessors :: 24
 ** SystemLoadAverage :: 0.49

Hopefully this is of use to you, or inspires you to investigate JRuby a bit further. I’d highly recommend it for Java shops who want to take advantage of Ruby’s scripting ability.

Mar

A DSL for collections in Java

by Joe Wright

When writing Java code I often find it laborious to create collections using the java.util.* collection classes. To avoid this, I’ve been using a mini-DSL to reduce my collections code.

import static com.joejag.common.collections.Dsl.*;

// A list
List list = list("abc", "def");

// A set
Set set = set("Sleepy", "Sneezy", "Dozy");

// A Map
Map map = map(entry("Joe", 28), entry("Gerry", 39));

Here is the underlying code.

package com.joejag.common.collections;

import java.util.*;

public class Dsl {
    public static <T> List<T> list(T... args) {
        return Arrays.asList(args);
    }

    public static <T> Set<T> set(T... args) {
        Set<T> result = new HashSet<T>(args.length);
        result.addAll(Arrays.asList(args));
        return result;
    }

    public static <K, V> Map<K, V> map(Entry<? extends K, ? extends V>... entries) {
        Map<K, V> result = new HashMap<K, V>(entries.length);

        for (Entry<? extends K, ? extends V> entry : entries)
            if (entry.value != null)
                result.put(entry.key, entry.value);

        return result;
    }

    public static <K, V> Entry<K, V> entry(K key, V value) {
        return new Entry<K, V>(key, value);
    }

    public static class Entry<K, V> {
        K key;
        V value;

        public Entry(K key, V value) {
            this.key = key;
            this.value = value;
        }
    }

    public static void main(String args[]) {
        List<String> list = list("abc", "def");
        System.out.println(list);

        Set<String> set = set("Sleepy", "Sneezy", "Dozy");
        System.out.println(set);

        Map<String, Integer> map = map(entry("Joe", 28), entry("Gerry", 39));
        System.out.println(map);
    }
}

I’ve noticed that the Google Collections project has morphed into Guava which has great reusable code for collections and a lot of other common Java tasks.

Mar

Start a web server from your pwd using Ruby and Thin

by Joe Wright

Occasionally I’m interested in grabbing files from a directory on a remote host. This could be for another process to consume or to look at on my local work station.

The standard way of doing this is using scp, ftp or a file share. I prefer to start a short-lived web server that shares the file system.

To make this simple I use a ruby script to allow me to start a webserver from the directory I’m currently in.

['rubygems', 'thin', 'rack', 'socket'].each {|file| require file }
Thin::Server.start(IPSocket.getaddress(Socket.gethostname), 7777) do
  use Rack::CommonLogger
  run Rack::Directory.new(Dir.pwd)
end

As an alias for ~/.bashrc it looks like this:

alias rshare="ruby -rubygems -e \"['thin', 'rack', 'socket'].each {|file| require file };"\
" Thin::Server.start(IPSocket.getaddress(Socket.gethostname), 7777) {"\
" use Rack::CommonLogger; run Rack::Directory.new(Dir.pwd) }\""

This allows you to go to port 7777 on the host and retrieve the files you’re interested in.

@joejag /tmp $ rshare
>> Thin web server (v1.2.7 codename No Hup)
>> Maximum connections set to 1024
>> Listening on 10.0.0.2:7777, CTRL+C to stop
10.0.0.8 - - [01/Mar/2011 07:45:15] "GET / HTTP/1.1" 200 2153 0.0038

You will need to have the ‘thin’ and ‘rack’ gems installed to do this.

Feb

Creating a simple Java RESTful service using Jersey and Maven

by Joe Wright

There has been a lot of interest in Resource Orientated Architecture (ROA) online which has resulted in an explosion of excellent libraries that make it simple to use this architectural style.

The example in this article uses Java, Maven and a JAX-RS library called Jersey to create a very simple orders system.

Our order consists of a reference number and a customer name. The system has three entry points:

HTTP PUT to create an order
HTTP GET to see the details of an order
HTTP GET to see all the existing orders

Download the full source of this article here.

To make it simple to get it up and running I’ve added configuration for Jetty to run on port 9090. You can compile the source code and run a web server to use it by issuing this single Maven command:

mvn jetty:run

I recommend that you use curl to test your RESTful web services as it is easy to change the HTTP verb that you are using. You can manipulate the orders system using these curl commands.

# Add a new order for Bob with ID 1
curl -X PUT http://localhost:9090/orders-server/orders/1?customer_name=bob 

# Check the status of the order
curl -X GET http://localhost:9090/orders-server/orders/1 

# See all the orders in the system
curl -X GET http://localhost:9090/orders-server/orders/list

Note: HTTP GET is the default verb used by curl, I’ve explicitly listed GET in the examples to make it clear which verb is in use.

The code

To use Jersey you need to add a servlet to your web.xml and create a resource implementation class.

web.xml

The WEB-INF/web.xml fragment sets up the Jersey Servlet with a parameter listing the package to search for RESTful classes.

<servlet>
   <servlet-name>Jersey Web Application</servlet-name>
   <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
   <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <!-- Important bit -->
      <param-value>com.joejag.code.orders.restservices</param-value>
   </init-param>
   <load-on-startup>1</load-on-startup>
</servlet>

This reads any classes in the com.joejag.code.orders.restservices and looks for annotations declaring resources.

The Java class

On the implementation class you can see the @Path annotation on the class signature indicating the resource we are handling. This is used by Jersey to configure the URL used to interact with the resource.

Each method has a sub-path declared, an HTTP verb to respond from and the response type to produce. The method parameters are bound by using either part of the path (with @PathParam) or a separate query parameter (with @QueryParam).

package com.joejag.code.orders.restservices;

import java.util.Map;
import java.util.TreeMap;

import javax.ws.rs.GET;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Response;

@Path("orders")
public class OrdersService
{
   // Stores state simply in a static collection class.
   private static Map<String, String> orders = new TreeMap<String, String>();

   @Path("/{order}")
   @PUT
   @Produces("text/html")
   public String create(@PathParam("order") String order,
                                    @QueryParam("customer_name") String customerName)
   {
      orders.put(order, customerName);
      return "Added order #" + order;
   }

   @Path("/{order}")
   @GET
   @Produces("text/html")
   public String find(@PathParam("order") String order)
   {
      if (orders.containsKey(order))
         return "<h2>Details on Order #" + order +
                    "</h2><p>Customer name: " + orders.get(order);

      throw new WebApplicationException(Response.Status.NOT_FOUND);
   }

   @Path("/list")
   @GET
   @Produces("text/html")
   public String list()
   {
      String header = "<h2>All Orders</h2>\n";

      header += "<ul>";
      for (Map.Entry<String, String> order : orders.entrySet())
         header += "\n<li>#" + order.getKey() + " for " + order.getValue() + "</li>";

      header += "\n</ul>";

      return header;
   }
}

Download the full source of this article here.

Where to learn more

There are a number of great articles on how to develop JAX-RS REST applications. I recommend you start with the Jersey guide for Java applications:

If you are using Ruby then take a look at the wonderful Sinatra project.

Jan

Renaming MP3 tags with Ruby and the mp3info gem

by Joe Wright

Occasionally you get MP3s which have unconventional tags. I usually get this problem with compilation albums and it’s a bit dull to rename them in iTunes.

There’s a ruby gem which let’s you easily edit id3 tags called ‘mp3info‘.

Here is a script to get you started:

require "mp3info"

dir = '/opt/music/album_name/'
Dir.entries(dir).each do |file|
   next if file !~ /.mp3$/ # skip files not ending with .mp3
   Mp3Info.open(dir + file) do |mp3|
      puts mp3.tag.title
      puts mp3.tag.artist
      puts mp3.tag.album
      puts mp3.tag.tracknum

      # You would perform your text transform here, here's a simple change instead
      mp3.tag.title = mp3.tag.title + ' with my change'
      mp3.tag.artist = mp3.tag.artist + ' with my change'
   end
end

Jan

Installing CouchDB 1.0.1 on RHEL4 as the local user

by Joe Wright

On some of the servers I use I don’t have root access. This means occasionally it can be a hassle to install packages when the dependencies aren’t available.

Recently I had to install CouchDB and it’s dependencies from source. The tricky bit was having to edit the CouchDB configure script to specify the Curl libraries.

In this example I’ve assumed the user home directory is ‘/home/your_user’, with the Couch and it’s dependencies ending up in ‘/home/your_user/db/couch/’.

# Erlang 5.6.5
wget http://www.erlang.org/download/otp_src_R12B-5.tar.gz
tar -zxvf otp_src_R12B-5.tar.gz
cd otp_src_R12B-5
./configure --prefix=/home/your_user/db/couch/erlang
make && make install

# SpiderMonkey 1.7.0
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.7.0.tar.gz
tar -zxvf js-1.7.0.tar.gz
cd js/src/
make -f Makefile.ref
make BUILD_OPT=1 JS_DIST=/home/your_user/db/couch/spidermonkey/ \
  -f Makefile.ref export

# ICU 4.6
wget http://download.icu-project.org/files/icu4c/4.6/icu4c-4_6-src.tgz
tar -zxvf icu4c-4_6-src.tgz
cd icu
./configure --prefix=/home/your_user/db/couch/icu
make && make install

# Curl 7.21.2
wget http://curl.haxx.se/download/curl-7.21.2.tar.gz
tar -zxvf curl-7.21.2.tar.gz
cd curl-7.21.2
./configure --prefix=/home/your_user/db/couch/curl
make && make install

# Add the programs to your $PATH
export PATH=$PATH:/home/your_user/db/couch/erlang/bin
export PATH=$PATH:/home/your_user/db/couch/spidermonkey/bin
export PATH=$PATH:/home/your_user/db/couch/icu/bin
export PATH=$PATH:/home/your_user/db/couch/curl/bin
export LD_LIBRARY_PATH=/home/your_user/db/couch/icu/lib

# CouchDB 1.0.1
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org//couchdb/1.0.1/apache-couchdb-1.0.1.tar.gz
tar -zxvf apache-couchdb-1.0.1.tar.gz
cd apache-couchdb-1.0.1
## edit configure.  Search for CURL_LDFLAGS=-lcurl.
## Replace with: CURL_LDFLAGS="-L/home/your_user/db/couch/curl/lib -lcurl"
./configure --prefix=/home/your_user/db/couch/couchdb \
  --with-js-lib=/home/your_user/db/couch/spidermonkey/lib \
  --with-js-include=/home/your_user/db/couch/spidermonkey/include \
  --with-erlang=/home/your_user/db/couch/erlang/lib/erlang/usr/include
make && make install

# Start up CouchDB
/home/your_user/db/couch/couchdb/bin/couchdb
# Time to Relax.
# From this command you should see: {"couchdb":"Welcome","version":"1.0.1"}
curl http://127.0.0.1:5984/

Jan

Adding colour to your unix prompt (PS1) and some handy ls aliases

by Joe Wright

This is my standard .bashrc that I use on all my boxes with a bash prompt. It gives you colourful listings when you use ‘ls’ and some shorthand versions of common ‘ls’ parameters.

The PS1 is set to just show the username, directory and the command in a brighter colour than the output from commands:

# ls aliases that use colour
alias la='ls -a --color=auto'
alias lla='ls -la --color=auto'
alias lt='ls -ltr --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias lsh='ls -lSh --color=auto'

# Unix prompt: /user /directory $
PS1='\[\e[0;32m\]\u\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$\[\e[m\] \[\e[1;37m\]'

# start a web server in this directory
alias rshare="ruby -rubygems -e \"['thin', 'rack', 'socket'].each {|file| require file };"\
" Thin::Server.start(IPSocket.getaddress(Socket.gethostname), 7777) {"\
" use Rack::CommonLogger; run Rack::Directory.new(Dir.pwd) }\""

Example in /bin:
ps1

Nov

103

New language features in Java 7

by Joe Wright

I’m just back from the Devoxx conference in Antwerp. An update was given on the new language changes that will be in Java 7. The JDK currently has a release date of September 2010.

Here are 7 of the new features that have been completed:

~~Language support for collections~~ ( Postponed to Java 8 )
Automatic Resource Management
Improved Type Inference for Generic Instance Creation (diamond)
Underscores in numeric literals
Strings in switch
Binary literals
Simplified Varargs Method Invocation

There is a lot more to Java 7 then just these language changes. I’ll be exploring the rest of the release in future posts. One of the big debates is currently around Closures, which are a separate JSR.

Language support for collections

This has been postponed to Java 8! You could use my simple alternative until then.

Java will be getting first class language support for creating collections. The style change means that collections can be created like they are in Ruby, Perl etc.

Instead of:

List<String> list = new ArrayList<String>();
list.add("item");
String item = list.get(0);

Set<String> set = new HashSet<String>();
set.add("item");

Map<String, Integer> map = new HashMap<String, Integer>();
map.put("key", 1);
int value = map.get("key");

You will be able to use:

List<String> list = ["item"];
String item = list[0];

Set<String> set = {"item"};

Map<String, Integer> map = {"key" : 1};
int value = map["key"];

These collections are immutable.

Automatic Resource Management

Some resources in Java need to be closed manually like InputStream, Writers, Sockets, Sql classes. This language feature allows the try statement itself to declare one of more resources. These resources are scoped to the try block and are closed automatically.

This:

BufferedReader br = new BufferedReader(new FileReader(path));
try {
   return br.readLine();
} finally {
   br.close();
}

becomes:

try (BufferedReader br = new BufferedReader(new FileReader(path)) {
   return br.readLine();
}

You can declare more than one resource to close:

try (
   InputStream in = new FileInputStream(src);
   OutputStream out = new FileOutputStream(dest))
{
 // code
}

To support this behaviour all closable classes will be retro-fitted to implement a Closable interface.

Improved Type Inference for Generic Instance Creation (diamond)

This is a particular annoyance which is best served with an example:

Map<String, List<String>> anagrams = new HashMap<String, List<String>>();

becomes:

Map<String, List<String>> anagrams = new HashMap<>();

This is called the diamond operator: <> which infers the type from the reference declaration.

Underscores in numeric literals

Long numbers are hard to read. You can now split them up using an underscore in ints and longs:

int one_million = 1_000_000;

Strings in switch

Currently you can only use numbers or enums in switch statements. String has been added as a candidate:

String s = ...
switch(s) {
 case "quux":
    processQuux(s);
    // fall-through

  case "foo":
  case "bar":
    processFooOrBar(s);
    break;

  case "baz":
     processBaz(s);
    // fall-through

  default:
    processDefault(s);
    break;
}

Binary literals

Java code, due to its C heritage, has traditionally forced programmers to represent numbers in only decimal, octal, or hexadecimal.

As quite a few domains are bit orientated, this restriction can introduce errors. You can now create binary numbers using an 0b prefix.

int binary = 0b1001_1001;

Simplified Varargs Method Invocation

When a programmer tries to invoke a *varargs* (variable arity) method with a non-reifiable varargs type, the compiler currently generates an “unsafe operation” warning. JDK 7 moves the warning from the call site to the method declaration. This will enable API designers to use varargs due to the reduction of warnings reported.

This one is slightly more involved so you are better off looking at the proposal.

Sep

Coding Dojo: Example Katas

by Joe Wright

In my previous post I discussed the basics of what a Coding Dojo is and why you’d want to have one.

I host a Coding Dojo at my work every month, so I’ve decided to periodically update an index page of example Katas. They are usually based off existing Katas which are available on the main Coding Dojo website. I’d also recommend the prolific Sao Paulo Coding Dojo.

You should be able to reuse the same problems at your own Dojo. If you have any other recommendations then please email/twitter me, or comment on this post.

Current Problems

Minesweeper Kata

On a slightly related note, I just completed the common Minesweeper Kata.

Matt Wynne, who has an excellent blog, recently created a test harness using the Ruby BDD framework called Cucumber. It acts as a great way to introduce you to BDD and Github if you haven’t used them before.

How to send a raw HTTP request via Java

Checking JMX with JRuby

A DSL for collections in Java

Start a web server from your pwd using Ruby and Thin

Creating a simple Java RESTful service using Jersey and Maven

The code

web.xml

The Java class

Where to learn more

Renaming MP3 tags with Ruby and the mp3info gem

Installing CouchDB 1.0.1 on RHEL4 as the local user

Adding colour to your unix prompt (PS1) and some handy ls aliases

New language features in Java 7

Language support for collections

Automatic Resource Management

Improved Type Inference for Generic Instance Creation (diamond)

Underscores in numeric literals

Strings in switch

Binary literals

Simplified Varargs Method Invocation

Coding Dojo: Example Katas

Current Problems

Minesweeper Kata

About

Categories