eScience and Data Science at the University of Washington eScience Institute
"Hangover" Keynote by Bill Howe, Director of Research, Scalable Data Analytics, eScience Institute Affiliate Assistant Professor, Department of Computer Science & Engineering, University of Washington
Scientific process getting reduced to database problem - instead of querying the world we download the world and query the database...
UoW eScience Inst to get in the forefront of research in eScience techniques/technology, and in fields that depend on them.
3Vs of big data:
volume - this gets lots of attention but
variety - this is the bigger challenge
velocity
Sources a longtail image from Carol Goble showing lots of data in Excel spreadsheets, lab books, etc, is just lost.
Types of data stored - especially data data and some text. 87% of time is on "my computer"; 66% a hard drive...
Mostly people are still in the gigabytes range, or megabytes, less so in terabytes (but a few in petabytes).
No obvious relationship between funding and productivity. Need to support small innovators, not just the science stars.
Problem - how much time do you spend handling data as opposed to doing science? General answer is 90%.
May be spending a week doing manual copy-paste to match data because not familiar with tools that would allow a simple SQL JOIN query in seconds.
Sloan Digital Sky Survey incredibly productive because they put the data online in database format and thousands of other people could run queries against it.
SQLShare: Query as a service
Want people to upload data "as is". Cloud-hosted. Immediately start writing queries, share results, others write their queries on top of your queries. Various access methods - REST API -> R, Python, Excel Addin, Spreadsheet crawler, VizDeck, App on EC2.
Metadata
Has been recommending throwing non-clean data up there. Claims that comprehensive metadata standards represent a shared consensus about the world but at the frontier of research this shared consensus by definition doesn't exist, or will change frequently, and data found in the wild will typically not conform to standards. So modifies Maslow's Needs Hierarchy:
Usually storage > sharing > curation > query > analytics
Recommends: storage > sharing > query > analytics > curation
Everything can be done in views - cleaning, renaming columns, integrating data from different sources while retaining provenance.
Bring the computation to the data. Don't want just fetch-and-retrieve - need a rich query service, not a data cemetary. "Share the soup and curate incrementally as a side-effect of using the data".
Convert scripts to SQL and lots of problems go away. Tested this by sending postdoc to a meeting and doing "SQL stenography" - real-time analytics as discussion went on. Not a controlled study - didn't have someone trying to do it in Python or R at same time - but would challenge someone to do it as quickly! Quotes (a student?) "Now we can accomplish a 10minute 100line script in 1 line of SQL." Non-programmers can write very complex queries rather than relying on staff programmers and feeling 'locked out'.
Data science
Taught an intro to data science MooC with tens of thousands of students. (Power of discussion forum to fix sloppy assignment!)
Lots of students more interested in building things than publishing, and are lost to industry. So working on 'incubator' projects, reverse internships pulling people back in from industry.
Q: Have you experimented with auto-generating views to cleanup?
A: Yes, but less with cleaning and more deriving schemas and recommending likely queries people will want. Google tool "Data wrangler".
Q: Once again people using this will think of themselves as 'not programmers' - isn't this actually a downside?
A: Originally humans wrote queries, then apps wrote queries, now humans are doing it again and there's no good support for development in SQL. Risk that giving people power but not teaching programming. But mostly trying to get people more productive right now.
Showing posts with label cloud computing. Show all posts
Showing posts with label cloud computing. Show all posts
Thursday, 4 July 2013
Tuesday, 2 July 2013
Design patterns for lab #labpatterns; Research cloud - #nzes
A pattern language for organising laboratory knowledge on the web #labpatterns
Cameron McLean, Mark Gahegan & Fabiana Kubke, The University of Auckland
Google Site
Lots of lab data hard to find/reuse - big consequences for efficiency, reproducibility, quality.
Want to help researchers locate, understand, reuse, and design data. Particularly focused on describing semantics of the experiment (rather than semantics of data).
Design pattern concept originated in field of architecture. Describes a solution to a problem in a context. Interested in idea of forces - essential/invariant concepts in a domain.
Kitchen recipe as example of design pattern for lab protocol.
What are recurring features and elements? Forces for a cake-like structure include: structure providers (flour), structure modifier (egg), flavours; aeration and heat transfer.
Apply this to lab science, in a linked science setting. Take a "Photons alive" design pattern (using light to virtualise biological processes in an animal). See example paper. Can take a sentence re methodology and annotate eg "imaging" as diagnostic procedure. This using current ontologies gives you the What but not the Why. Need to tag with a "Force" concept eg "immobilisation". Deeper understanding of process - with role of steps. And can start thinking about what other methods of immobilisation there may be.
So how can we make these patterns? Need to use semantic web methods.
A wiki for lab semantics. (Wants to implement this.) Semantic form on wiki - a template. Wiki serves for attribution, peer review, publication - and endpoint to RDF store.
Q: How easy is this to use for a domain expert?
A: Semantic modeling is iterative process and not easy. But semantic wiki can hide complexity from enduser so domain expert can just enter data.
Q: We spend lots of time pleading with researchers to fill out webforms. How else can we motivate them, eg to do it during process rather than at end?
A: Certain types of people are motivated to use wiki. This is first step, proof of concept. Need a critical mass before self-sustaining.
Q: How much use would this actually be for domain experts? Would people without implicit knowledge gain from it?
A: Need to survey this and evaluate. It's valuable as a democratising process.
Q: What about patent/commercial knowledge?
A: Personally taking Open science / linked science approach - intended for research that's intended to be maximally shared.
A "Science Distribution Network" - Hadoop/ownCloud syncronised across the Tasman
Guido Aben, AARNet; Martin Feller, The University of Auckland; Andrew Farrell, New Zealand eScience Infrastructure; Sam Russell, REANNZ
Have preferred to do one-to-few applications rather than google-style one-to-billions. Now changing. Because themselves experiencing trouble sending large files. Scraped up own file transfer system, marketed as cloudstor though not in the cloud and doesn't store things. Expected couple hundred uses, got 6838 users over the last use. Why linear growth? "Apparently word of mouth is a linear thing..." Seem to be known by everyone who have file-sharing issues.
FAQs:
Can we keep files permanently?
Can I upload multiple files?
Why called cloudstor when it's really for sending?
"cloudstor+ beta" - looks like dropbox so why doing this if already there? They're slow (hosted in Singapore or US). Cloudstor+ 30MB/s cf 0.75MB/s as a maximum for other systems. Pricing models not geared towards large datasets. And subject to PRISM etc.
Built on a stack:
Anycast | AARNet
ownCloud - best OSS they've seen/tested so far - has plugin system and defined APIs
MariaDB
hadoop - but looking at substituting with XTREEMFS which seems to work with latencies.
Distributed architecture - can be extended internationally. Would like one in NZ, Europe, US, then scale up.
Bottleneck is from desktop to local node. Only way they can address this is to get as close to researcher as possible - want to build local nodes on campus.
Cameron McLean, Mark Gahegan & Fabiana Kubke, The University of Auckland
Google Site
Lots of lab data hard to find/reuse - big consequences for efficiency, reproducibility, quality.
Want to help researchers locate, understand, reuse, and design data. Particularly focused on describing semantics of the experiment (rather than semantics of data).
Design pattern concept originated in field of architecture. Describes a solution to a problem in a context. Interested in idea of forces - essential/invariant concepts in a domain.
Kitchen recipe as example of design pattern for lab protocol.
What are recurring features and elements? Forces for a cake-like structure include: structure providers (flour), structure modifier (egg), flavours; aeration and heat transfer.
Apply this to lab science, in a linked science setting. Take a "Photons alive" design pattern (using light to virtualise biological processes in an animal). See example paper. Can take a sentence re methodology and annotate eg "imaging" as diagnostic procedure. This using current ontologies gives you the What but not the Why. Need to tag with a "Force" concept eg "immobilisation". Deeper understanding of process - with role of steps. And can start thinking about what other methods of immobilisation there may be.
So how can we make these patterns? Need to use semantic web methods.
A wiki for lab semantics. (Wants to implement this.) Semantic form on wiki - a template. Wiki serves for attribution, peer review, publication - and endpoint to RDF store.
Q: How easy is this to use for a domain expert?
A: Semantic modeling is iterative process and not easy. But semantic wiki can hide complexity from enduser so domain expert can just enter data.
Q: We spend lots of time pleading with researchers to fill out webforms. How else can we motivate them, eg to do it during process rather than at end?
A: Certain types of people are motivated to use wiki. This is first step, proof of concept. Need a critical mass before self-sustaining.
Q: How much use would this actually be for domain experts? Would people without implicit knowledge gain from it?
A: Need to survey this and evaluate. It's valuable as a democratising process.
Q: What about patent/commercial knowledge?
A: Personally taking Open science / linked science approach - intended for research that's intended to be maximally shared.
A "Science Distribution Network" - Hadoop/ownCloud syncronised across the Tasman
Guido Aben, AARNet; Martin Feller, The University of Auckland; Andrew Farrell, New Zealand eScience Infrastructure; Sam Russell, REANNZ
Have preferred to do one-to-few applications rather than google-style one-to-billions. Now changing. Because themselves experiencing trouble sending large files. Scraped up own file transfer system, marketed as cloudstor though not in the cloud and doesn't store things. Expected couple hundred uses, got 6838 users over the last use. Why linear growth? "Apparently word of mouth is a linear thing..." Seem to be known by everyone who have file-sharing issues.
FAQs:
Can we keep files permanently?
Can I upload multiple files?
Why called cloudstor when it's really for sending?
"cloudstor+ beta" - looks like dropbox so why doing this if already there? They're slow (hosted in Singapore or US). Cloudstor+ 30MB/s cf 0.75MB/s as a maximum for other systems. Pricing models not geared towards large datasets. And subject to PRISM etc.
Built on a stack:
Anycast | AARNet
ownCloud - best OSS they've seen/tested so far - has plugin system and defined APIs
MariaDB
hadoop - but looking at substituting with XTREEMFS which seems to work with latencies.
Distributed architecture - can be extended internationally. Would like one in NZ, Europe, US, then scale up.
Bottleneck is from desktop to local node. Only way they can address this is to get as close to researcher as possible - want to build local nodes on campus.
Wednesday, 14 April 2010
Mobile vs Smartphones & other links of interest 14/4/10
Mobile vs Smartphones
Roy Tennant suggests not making any more mobile websites as research suggests more people (in the US) are getting smartphones that can support anything a normal web-browser can support. (Though I don't know of any smartphone that supports a 1024x768 screensize...) Smartphone applications seem to be trending instead. The iLibrarian rounds up her Top 30 Library iPhone Apps (part 2 and part 3). Why an application when you've already got a website? Phil Windley points out that "If my bank can get me to download an app, then they have a permanent space on my app list." The trade-off is that whereas a website should work on any browser, smartphone apps often need to be in proprietary formats (the Librarian in Black particularly complains about Apple's iPhone in this respect).
Web 2.0
Common Craft has a 3-minute video explaining "Cloud Computing in Plain English".
The Metropolitan Museum of Art Libraries and Brown University Library provide a "dashboard" of widgets on their websites displaying current statistics about library usage.
View from the top :-)
The University Librarian at McMaster University Library blogs results from their laptop survey. Apparently laptop circulation now accounts for about a third of their total circulation stats; their survey looks into how students are using the laptops.
The Director of Librarys at the State University of New York at Potsdam blogs about "What I've Learned" in the first 10 months of her job there.
Scandal of the week...
Barbara Fister summarises recent discussion about EBSCO as the "New Evil Empire" in her Library Journal article "Big vendor frustrations, disempowered librarians, and the ends of empire".
Fun
Alice for the iPad - one of the ways technology can enhance the book.
Roy Tennant suggests not making any more mobile websites as research suggests more people (in the US) are getting smartphones that can support anything a normal web-browser can support. (Though I don't know of any smartphone that supports a 1024x768 screensize...) Smartphone applications seem to be trending instead. The iLibrarian rounds up her Top 30 Library iPhone Apps (part 2 and part 3). Why an application when you've already got a website? Phil Windley points out that "If my bank can get me to download an app, then they have a permanent space on my app list." The trade-off is that whereas a website should work on any browser, smartphone apps often need to be in proprietary formats (the Librarian in Black particularly complains about Apple's iPhone in this respect).
Web 2.0
Common Craft has a 3-minute video explaining "Cloud Computing in Plain English".
The Metropolitan Museum of Art Libraries and Brown University Library provide a "dashboard" of widgets on their websites displaying current statistics about library usage.
View from the top :-)
The University Librarian at McMaster University Library blogs results from their laptop survey. Apparently laptop circulation now accounts for about a third of their total circulation stats; their survey looks into how students are using the laptops.
The Director of Librarys at the State University of New York at Potsdam blogs about "What I've Learned" in the first 10 months of her job there.
Scandal of the week...
Barbara Fister summarises recent discussion about EBSCO as the "New Evil Empire" in her Library Journal article "Big vendor frustrations, disempowered librarians, and the ends of empire".
Fun
Alice for the iPad - one of the ways technology can enhance the book.
Subscribe to:
Posts (Atom)