Server-side logging for Apache Tomcat

Okay, so after spending two days on a problem that wasn't an actual problem, I was frustrated and determined that the only way I could get any feedback from the program was to implement a logging. I don't know if I've never had to use logging before because I'm smart and thus don't need it, or because I'm stupid and like doing things the hard way, but there goes.

A very simple guide to using logging for Apache Tomcat:

1.) The output will be put into the logs in the tomcat directory catalina.20XX-XX-XX.log.

2.) Include the following imports:
import java.util.logging.Level;
import java.util.logging.Logger;

3.) Initialize the logger in the class you want logged:
private static String nameOfLogger = <INSERT_CLASS_NAME_HERE>.class.getName();
private static Logger myLogger = Logger.getLogger(nameOfLogger); '

4.) Insert the following code if you want something always to be logged (alternatively, you can change "severe" to other levels, if you want the messages to only be printed if a certain level is reached):
myLogger.severe(<STRING YOU WANT PRINTED>);

5.) Don't forget to set the level of the logger:
myLogger.setLevel(Level.WARNING);

6.) Also create a logging.properties file in the /war/WEB-INF/ folder
An example file is like this:
# A default java.util.logging configuration.
# (All App Engine logging is through java.util.logging by default).
#
# To use this configuration, copy it into your application's WEB-INF
# folder and add the following to your appengine-web.xml:
#
# <system-properties>
#   <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
# </system-properties>
#

# Set the default logging level for all loggers to WARNING
.level = WARNING

For more information check out this wonderful guide

Integrating Open Reconcile with Google Refine

Since I published a WAR of open reconcile, I figure I should add a note on what to do if you want to have better integration between Google Refine and Open Reconcile (and have Google Refine's source code).

There are just two files you need to modify to add a "Configure Open Reconcile" option to the "Start reconciling" dialog box. First off, do know that I am sacrificing some FreeBase integration functionality. If this bothers you, don't follow these instructions. Also, if editing javascript/html scares you (why are you at this blog?), don't do this.

We're going to turn the "Add namespace" button into the button we want. It's rather simple, just open recon-dialog.html from grefine/main/webapp/modules/core/scripts/reconciliation/ Just change the text of the button from "Add namespace" to "Configure Open Reconcile".

Next, you'll undoubtedly see it calls a function. The function can be found in the same folder in recon-dialog.js. Find the function _addNamespacedService. Comment out or delete all of the existing code in that function (keeping in mind that it's not well-tabbed) and put this in instead:

     var self = this;
      var dialog = $(DOM.loadHTML("core", "scripts/reconciliation/config-open-service-dialog.html"));
      var elmts = DOM.bind(dialog);

      var level = DialogSystem.showDialog(dialog);
      var dismiss = function() {
        DialogSystem.dismissUntil(level - 1);
      };

      elmts.cancelButton.click(dismiss);
      elmts.addButton.click(function() {
        var url = $.trim(elmts.input[0].value);
        if (url.length > 10) {
            window.open(url.substring(0,url.length-10));
        }
        dismiss();
      });
      elmts.input.focus().select();

If you're paying attention, you will see that I'm referencing an html file that doesn't exist. I just coped add-standard-service-dialog.html and changed    
 <button class="button" bind="addButton">Add Service</button>
to
<button class="button" bind="addButton">Go To Config</button>

Okay, just return the file or re-create distribution files if you don't run it in the console (see <a href="http://code.google.com/p/google-refine/wiki/DevelopersGuide">Google Refine documentation</a> for how to do that).

I find this is helpful so I don't have to keep links to everything. It takes the url in, chops of the "/reconcile" that points to the service, and in this webapp the configuration screen is at the root. To do most options (Delete a row, preview data the row will pull, and eventually and substitution rules or synonyms) you need to "List All Current Libraries".

OpenReconcile

The project is almost done. I'm really tempted to add a synonym table function, where you can specify and type and have it automatically replace one value with another if the criteria is met. It'd be useful for things like species name, where human is often entered in place of homo sapiens.

Of course I'd also like to implement learning too, but that's not going to happen this semester.

Finally!

Okay, after a long period of constantly switching gears, I have a product that works that is open-source-able, but will meet my employer's needs out of the box without any customization!

It allows the user to pick between Oracle or PostGRES databases for their data. You can add information via a walkthru or just all by hand. The only thing that I think might be helpful is a test-feature, to make sure the connection can be made property. I think I'll try to get that added, and it'll fail if the information throws an error.

In other news, I finally have an example thesis and know what I need to do to finish (write 100 pages, lol). Also random thing I learned: San Francisco is one of the most stressful cities in the U.S., very true. So much wealth and poverty, too many extremes.

Open Reconciliation

I've completed opening the reconciliation service. Now it has a front-page that lets the user intelligently put in references to where s/he wants data pulled from. The data is then stored in a SQLite database, which the servlet reads from the populate the metadata and to grab the vocabulary that the requests are reconciled against.

It will be put up on: http://code.google.com/p/open-reconcile/ when I have it looking less jury-rigged.

SQLite and Moving

First off, after successfully adapting Google Refine to my company's needs, I now go full circle and adapt the software I wrote for the company to be open-source and useable by all. This task has lead to met implementing server-side SQLite using GWT framwork (and the sqlitejdbc JAR). It's going surprisingly well. It's probably not the best most secure idea in the world, but at least where I am, it'll be behind a log-in screen.

After this I may get involved in contributing to Google Refine's development. It depends on my company's needs, of course. All and all, this semester has been one of the most productive of my life.

Rebranding Google Refine

Google Refine is cool with you modifying their code and redistributing it... as long as you don't call it "Google Refine." So, I'm going through the process of rebranding Google Refine as BIORefine.

The first thing I did was whip up a nifty little replacement for the Google Refine logo.

It was created with two different heights 30px and 40px. it goes in the grefine/main/webapp/modules/core/images folder.

grefine/main/webapp/modules/core/index.vt is edited to change Google Refine to BIORefine.

There are a handful of other files to be changed (mostly the .html files in grefine/main/webapp/modules/core/scripts/index/ such as create-project-ui-source-selection.html).

Edit: I'll probably update this guide as I find more that I've missed (like about.html - although I did leave all the text in, just added that BIORefine is a modified version of Google Refine).

Update

So, it looks like my proposal for my project will finally be approved soon. :-) Also the project is pretty much done. I have it good to go as far as Richard XXXX is concerned about generating sensible matches. Part of me would like to play and implement some sort of Needleman&Wunsch algorithm instead of edit distance, but... it's not a huge deal, and I don't think I'd get that much more out of it.

I look forward to seeing how it performs on Wednesday, when I'm on campus, and don't have to do the vpn-thing. It should perform much better - the cell names took the longest, but that's because there are thousands of them, and they're much more likely to have weird punctuation issues. Two hours to check a couple thousand, but then again, it beats checking them all by hand by a huge amount.

I took out the suggestion request code, it's just not something I see as valuable for the service to do. There's going to be enough overlap between tissue type and disease that it may be confusing and the user should know what they're looking for.

Updates and more thoughts

I added a nearest neighbor algorithm to find misspellings. I think I'm going to take out some of the pattern matchings and just use the nearest neighbor scoring. It seems to give better results, and when terms are very short, it'll work better. (A misgiven id name of "C2" getting matched to a hunderd different things that are very long isn't that useful).

I want to find out if I can have the reconciliation happen in an autogenerated new column, for the purpose of building a synonym table. 

And I really hate mold, I feel so sick.

Farewell to Freebase in Google Refine

To delete Freebase I:

- deleted extensions/freebase folder.

- edited extensions/build.xml to remove <ant dir="freebase/" target="build" />

- edited main/webapp/module/core/scripts/reconiliation/recon-manager.js and removed
 ReconciliationManager.customServices.push({
     "name" : "Freebase Query-based Reconciliation",
     "ui" : { "handler" : "ReconFreebaseQueryPanel" }
   });

 and changed:

      ReconciliationManager.registerStandardService(
            "http://4.standard-reconcile.dfhuynh.user.dev.freebaseapps.com/reconcile");
      }

        to:

        ReconciliationManager.registerStandardService(
            "http://my service/reconcile");
      }

- edited main/webapp/module/core/scripts/project/exporters.js to remove the MSQL and TripleWriter options.

Project is progressing well!

I've gotten Freebase disintegrated from Google Refine as much as possible for now. I was to explore a few other last questions I have about some of the freebase references that are coded into various classes. I'm working on a reconciliation service configuration site that will allow users to add sources of data to the reconciliation service. It should be fun.

Milestone

I've reached a huge milestone. The service works with Google Refine now. This is excellent and fantastic and everything peachy because it enables me to run test cases so I can ensure it behaves as it should. Right now there's only one additional feature I want to add. It's done as far as I'm concerned. Next: Take away Freebase extension, see if I can hard-code it to look for the company's service only.

Progress!

The single query works well. The multiple query is... just being difficult, but it should resolve itself soon. Next to do: Add type-based help for matching. Take out Freebase stuff. There's no reason why anyone needs to use the Freebase features at my company. I should have this all done in a little less than a month's time. :-)

Reconciliation Service Tomcat Installation Notes

Note to self: class12.jar needs to be in the lib directory. After it's put in there tomcat needs to be restarted, so that it'll be added to the classpath. Now to work on matching/scoring for the single item query. Also - remember that multiple queries aren't working yet, so stop trying to run that!!! I'm runninng on maybe 1 hr of sleep, so I'm a little loopy today. Yeay end of semester time combined with bad health that makes working on stuff harder (who doesn't love having a repetitive stress injury that is just getting worse by doing anything related to everything I need to do?).

Progress and To-Do

I have the GSON for the results working for the single query, kind of. Right now I just have it able to match a single result. I need to create a class that is an array of them. I need to figure out how I want to handle multiple queries results (these are much more common in practical use).

JSON versus JSONP with a GSON example

JSON is a data format used for passing data between webapps.
An example of JSON
{
"name" : "Freebase Reconciliation Service",
"identifierSpace" : "http://rdf.freebase.com/ns/type.object.mid",
"schemaSpace" : "http://rdf.freebase.com/ns/type.object.id",
"view" : {
"url" : "http://www.freebase.com/view{{id}}"
},
"preview" : {
"url" : "http://www.freebase.com/widget/topic{{id}}?mode=content",
"width" : 430,
"height" : 300
},
"suggest" : {
"property" : "http://standard-reconcile.freebaseapps.com/suggest_property"
},
"defaultTypes" : [
{
"id" : "/people/person",
"name" : "Person"
},
{
"id" : "/location/location",
"name" : "Location"
}
 ]
}

JSON is meant to be used only within the same domain, a domain of origin thing to prevent its from being vulnerable to cross site scripting, like what was exploited like crazy when I was a young one just learning Java. But that's not always a very convenient solution for everyone, sometimes you just need to get JSON-based data from a 3rd party site (after all JSON has advantages over other methods), so what to do?

JSONP was developed to solve this. JSONP looks almost entirely just like JSON, even in name. An example of JSONP where "foo" is added to the call in form of a callback value
(i.e. http://www.mydomain.com/mywebapp?callback=foo):
foo({
  "name" : "Freebase Reconciliation Service",
  "identifierSpace" : "http://rdf.freebase.com/ns/type.object.mid",
  "schemaSpace" : "http://rdf.freebase.com/ns/type.object.id",
  "view" : {
    "url" : "http://www.freebase.com/view{{id}}"
  },
  "preview" : {
    "url" : "http://www.freebase.com/widget/topic{{id}}?mode=content",
    "width" : 430,
    "height" : 300
  },
  "suggest" : {
    "property" : "http://standard-reconcile.freebaseapps.com/suggest_property"
  },
  "defaultTypes" : [
    {
      "id" : "/people/person",
      "name" : "Person"
    },
    {
      "id" : "/location/location",
      "name" : "Location"
    }
  ]
})

So if you're like me, and thinking GSON is neat and you want to use it, but need the data to be packaged in JSONP format, it's rather easy to fix this..

   // Okay, it doesn't have to have the setPrettyPrinting,
  // but it helps make it readable when troubleshooting!

   Gson gson = new GsonBuilder().setPrettyPrinting().create();;
    response.setCharacterEncoding("UTF-8");
                 response.setContentType("application/json");
           response.setStatus(200);

   PrintWriter out= response.getWriter();
   String JSONstr = gson.toJson(object2beJSONized);
   String callback = request.getParameter("callback");
   JSONstr = callback+"("+JSONstr+");";
   out.print(test);

Ta-da~!

I'm posting this because I really wish there had been this easy of an explanation for how to do this somewhere online.

To do for today

Get reconciliation service working - apparently Google Refine seems to make POST requests... I had it programmed focused on GET oops! It looks like callbacks may be the issue, see how the vitro system handles them. Google Refine 2.5 is nice, and I have the code checked out for it. It allows data from clipboard to be input, for some odd reason I think that it is a plus to have that.

More thoughts...

To do: Get rid of namespace reconciliation feature - it's entirely Freebase-based. Strip down features to what is useful per Rich (dif. between cell operations and column operations confusing). Good news! - Reconciliation service on tomcat is able to spit out metadata all happy-like. It shouldn't be too bad to extend, but I want to wait until everything else is done first.

New Ref

http://sourceforge.net/apps/mediawiki/vivo/index.php?title=Extending_Google_Refine_for_VIVO --- this project is similar yet markedly different than mine.

To do lists

To do: Highest priority: implement other vocab rec. services. 2nd: take all the freebase options out of the GUI. 3rd: start migration over to Tomcat server. Wish List: Link matched vocab terms up to pages with information about the term (our definition) Create some way for a request to be sent to the appropriate person when person adds new term? VERY HIGH PRIORITY: Meet w/ end-users to develop not-self generated use-case. Personal To do: 1.) Get Systems Core requirement figured out. 2.) Work on form. 3.) Meet with appropriate prof to finish paperwork. 4.) Print out papers regarding term projects.