How to send a raw HTTP request via Java

While trying to figure out how a service worked I recently had to put together a Java class that let you easily replay a http conversation which had been sniffed.

I got bored of using telnet while making small changes to the payload and curl wants requests converted into an XML format it appears.

This simple socket based class lets you send a captured http request to a service. I’ve removed the exception handling for brevity.

import java.io.*;
import java.net.Socket;
import java.util.ArrayList;
import java.util.List;

public class Client {

    public static void main(String[] args) throws IOException {
        Socket socket = new Socket(args[0], 80);

        BufferedWriter out = new BufferedWriter(
                new OutputStreamWriter(socket.getOutputStream(), "UTF8"));
        BufferedReader in = new BufferedReader(
                new InputStreamReader(socket.getInputStream()));

        sendMessage(out, new File(args[1]));
        readResponse(in);

        out.close();
        in.close();
    }

    private static void sendMessage(BufferedWriter out, File request) throws IOException {
        System.out.println(" * Request");

        for (String line : getContents(request)) {
            System.out.println(line);
            out.write(line + "\r\n");
        }

        out.write("\r\n");
        out.flush();
    }

    private static void readResponse(BufferedReader in) throws IOException {
        System.out.println("\n * Response");

        String line;
        while ((line = in.readLine()) != null) {
            System.out.println(line);
        }
    }

    private static List getContents(File file) throws IOException {
        List contents = new ArrayList();

        BufferedReader input = new BufferedReader(new FileReader(file));
        String line;
        while ((line = input.readLine()) != null) {
            contents.add(line);
        }
        input.close();

        return contents;
    }
}

For example. Running this class with the parameters: “google.co.uk /path/to/stored/file” with the stored file being just:

GET /intl/en/policies/privacy/ HTTP/1.1

Will give you this output:

 * Sending
GET /intl/en/policies/privacy/ HTTP/1.1

 * Response
HTTP/1.1 200 OK
Vary: Accept-Encoding
Content-Type: text/html
Last-Modified: Fri, 27 Jan 2012 17:53:03 GMT
Date: Tue, 07 Feb 2012 21:40:30 GMT
Expires: Tue, 07 Feb 2012 21:40:30 GMT
Cache-Control: private, max-age=0
X-Content-Type-Options: nosniff
Server: sffe
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked

// Body content of web page

For capturing payloads I recommend using tcpflow running on the target host as “tcpflow port 80″

Renaming MP3 tags with Ruby and the mp3info gem

Occasionally you get MP3s which have unconventional tags. I usually get this problem with compilation albums and it’s a bit dull to rename them in iTunes.

There’s a ruby gem which let’s you easily edit id3 tags called ‘mp3info‘.

Here is a script to get you started:

require "mp3info"

dir = '/opt/music/album_name/'
Dir.entries(dir).each do |file|
   next if file !~ /.mp3$/ # skip files not ending with .mp3
   Mp3Info.open(dir + file) do |mp3|
      puts mp3.tag.title
      puts mp3.tag.artist
      puts mp3.tag.album
      puts mp3.tag.tracknum

      # You would perform your text transform here, here's a simple change instead
      mp3.tag.title = mp3.tag.title + ' with my change'
      mp3.tag.artist = mp3.tag.artist + ' with my change'
   end
end

Installing CouchDB 1.0.1 on RHEL4 as the local user

On some of the servers I use I don’t have root access. This means occasionally it can be a hassle to install packages when the dependencies aren’t available.

Recently I had to install CouchDB and it’s dependencies from source. The tricky bit was having to edit the CouchDB configure script to specify the Curl libraries.

In this example I’ve assumed the user home directory is ‘/home/your_user’, with the Couch and it’s dependencies ending up in ‘/home/your_user/db/couch/’.

# Erlang 5.6.5
wget http://www.erlang.org/download/otp_src_R12B-5.tar.gz
tar -zxvf otp_src_R12B-5.tar.gz
cd otp_src_R12B-5
./configure --prefix=/home/your_user/db/couch/erlang
make && make install

# SpiderMonkey 1.7.0
wget http://ftp.mozilla.org/pub/mozilla.org/js/js-1.7.0.tar.gz
tar -zxvf js-1.7.0.tar.gz
cd js/src/
make -f Makefile.ref
make BUILD_OPT=1 JS_DIST=/home/your_user/db/couch/spidermonkey/ \
  -f Makefile.ref export

# ICU 4.6
wget http://download.icu-project.org/files/icu4c/4.6/icu4c-4_6-src.tgz
tar -zxvf icu4c-4_6-src.tgz
cd icu
./configure --prefix=/home/your_user/db/couch/icu
make && make install

# Curl 7.21.2
wget http://curl.haxx.se/download/curl-7.21.2.tar.gz
tar -zxvf curl-7.21.2.tar.gz
cd curl-7.21.2
./configure --prefix=/home/your_user/db/couch/curl
make && make install

# Add the programs to your $PATH
export PATH=$PATH:/home/your_user/db/couch/erlang/bin
export PATH=$PATH:/home/your_user/db/couch/spidermonkey/bin
export PATH=$PATH:/home/your_user/db/couch/icu/bin
export PATH=$PATH:/home/your_user/db/couch/curl/bin
export LD_LIBRARY_PATH=/home/your_user/db/couch/icu/lib

# CouchDB 1.0.1
wget http://mirrors.ukfast.co.uk/sites/ftp.apache.org//couchdb/1.0.1/apache-couchdb-1.0.1.tar.gz
tar -zxvf apache-couchdb-1.0.1.tar.gz
cd apache-couchdb-1.0.1
## edit configure.  Search for CURL_LDFLAGS=-lcurl.
## Replace with: CURL_LDFLAGS="-L/home/your_user/db/couch/curl/lib -lcurl"
./configure --prefix=/home/your_user/db/couch/couchdb \
  --with-js-lib=/home/your_user/db/couch/spidermonkey/lib \
  --with-js-include=/home/your_user/db/couch/spidermonkey/include \
  --with-erlang=/home/your_user/db/couch/erlang/lib/erlang/usr/include
make && make install

# Start up CouchDB
/home/your_user/db/couch/couchdb/bin/couchdb
# Time to Relax.
# From this command you should see: {"couchdb":"Welcome","version":"1.0.1"}
curl http://127.0.0.1:5984/

Adding colour to your unix prompt (PS1) and some handy ls aliases

This is my standard .bashrc that I use on all my boxes with a bash prompt. It gives you colourful listings when you use ‘ls’ and some shorthand versions of common ‘ls’ parameters.

The PS1 is set to just show the username, directory and the command in a brighter colour than the output from commands:

# ls aliases that use colour
alias la='ls -a --color=auto'
alias lla='ls -la --color=auto'
alias lt='ls -ltr --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias lsh='ls -lSh --color=auto'

# Unix prompt: /user /directory $
PS1='\[\e[0;32m\]\u\[\e[m\] \[\e[1;34m\]\w\[\e[m\] \[\e[1;32m\]\$\[\e[m\] \[\e[1;37m\]'

# start a web server in this directory
alias rshare="ruby -rubygems -e \"['thin', 'rack', 'socket'].each {|file| require file };"\
" Thin::Server.start(IPSocket.getaddress(Socket.gethostname), 7777) {"\
" use Rack::CommonLogger; run Rack::Directory.new(Dir.pwd) }\""

Example in /bin: