Comments on: FlightCaster merges its statistical-learning code into Incanter

By: More musings on MapReduce and bioinformatics

More musings on MapReduce and bioinformatics — Tue, 29 Dec 2009 06:15:29 +0000

[...] need the ability to abstract out the MapReduce part itself. Which is why Pig, Cascading, Incanter/Crane, or life science specific frameworks in the mould of Incanter/Crane will be [...]

By: liebke

liebke — Fri, 04 Dec 2009 20:58:39 +0000

Thanks, I look forward to hearing more about your startup in the future!

By: Jan Rychter

Jan Rychter — Fri, 04 Dec 2009 14:30:41 +0000

This is some very good stuff. I was already impressed by Incanter and this addition is exactly in line with what we’ll be doing soon (also a startup using machine learning technologies).

By: liebke

liebke — Fri, 04 Dec 2009 05:46:00 +0000

Thanks Chetan, that is great to hear!

David

By: Chetan Conikee

Chetan Conikee — Fri, 04 Dec 2009 05:36:18 +0000

Awesome Work David …. Glad to acknowledge that Bradford has jumped on the incanter bandwagon too. I have put a few of your libraries to work in production application (in the financial domain) and can’t say how happy and satisfied I am .
:)

By: liebke

liebke — Fri, 04 Dec 2009 04:31:43 +0000

Haha, my work on the book has been a bit delayed by the recent work merging FlightCaster’s code, but I hope to get back to work on it. I do think the recent work may change the emphasis a bit, we’ll see.

David

By: liebke

liebke — Fri, 04 Dec 2009 03:58:13 +0000

I have long planned to integrate Weka, and I would still like that to happen. I think the current FlightCaster-based approach has a different emphasis than Weka does. Right now we’re focusing on creating a lower-level toolkit for building custom statistical-learning algorithms that can be embedded in larger systems, including distributed systems. Weka’s approach is to provide a lot of great pre-built learning algorithms, but this approach provides less flexibility than systems like FlightCaster require. However, I think this different emphasis means there’s room for both approaches in Incanter, and I hope to include Weka integration in the future.

David

By: DM

DM — Fri, 04 Dec 2009 03:38:40 +0000

Forgot to say fantastic news though….thanks Flightcaster.

By: DM

DM — Fri, 04 Dec 2009 03:37:08 +0000

Sounds like a fair bit of overlap with Weka.
Would it be worthwhile to integrate Weka and Clojure. I think there’s an R to Weka package. Obvously incanted can link directly to Weka since it’s 100% java, more easily than R. Is it worth building incanter wrappers to Weka a bit like Colt etc.

By: liebke

liebke — Thu, 03 Dec 2009 23:03:53 +0000

I'm hoping Bradford and I will each blog more in the future on the new libraries. In general, the new functionality is based on FlightCaster's low-level statistical-learning toolkit, which is used to develop their custom analytics. We plan on developing higher-level functions based on this foundation in the future. The following are links to the relevant API documentation. We plan to improve all the docs, and provide more example usage. The following existing Incanter libraries have additional functions: incanter.stats: a bunch of new functions, including many distance and similarity metrics, incanter.io: more io functions and these are entirely new libraries: incanter.transformations incanter.probability incanter.information_theory incanter.incremental-stats incanter.classification incanter.chrono: a Joda time based library This is just the beginning, we plan on a lot more development in the coming months. David