When you read or hear the term "Big Data" what do you think about? Do you visualize an extremely large database or data warehouse? Do you think about architectures and algorithms for storing and processing big data? When haven’t we had big data in relation to the systems and services that we are asked to program? Is yesterday’s big data small data today? With the Internet and the number of humans creating and consuming data, I guess we do have really big data today and it is only getting bigger. When I see a temperature rise in the number of articles, blog posts, conferences and books that appear on a topic, I love to dig into the topic.
Here are a number of links to articles that I’ve been reading about big data:
- What is "Big Data?" - ZDNet
- The Pathologies of Big Data - ACM Queue
- Big Data to Drive a Surveillance Society - ComputerWorld
- Business Information consumption: 9,570,000,000,000,000,000,000 Bytes Per Year (9.57 zettabytes) - USCD
- Big Data processing with Hadoop - NotFluffJustStuff.com
- A free Visual Programming Language for Big Data - ReadWriteWeb.com
- Big Data for Science Workshop - Indiana University
- A Conversation with IBM’s Mr. Big Data - Infoworld
- Strata 2011 - O’Reilly’s Strata Making Data Work Conference
- Thoughts on the O’Reilly 2011 Strata Big Data Conference - Business Intelligence from the Swamp
- Big Data - Wikipedia
- Group Seeks Global Protocol to Identify Big Data Sets - Science Magazine
- Expert Panel: What’s Around the Bend for Big Data? - HP InTheCloud
To paraphrase some lines from the Beatles song "It’s Getting Better All The Time" (apologies to John, Paul, George and Ringo):
I used to get mad at my data
The databases I used weren’t cool
You data has grown, relations complex
We might partially blame George Boole.
I’ve got to admit data’s getting bigger
A little bigger all the time
I have to admit data’s getting bigger
It’s getting bigger since you’ve been online.
How big is your data? Is it getting even bigger? What architectures and tools are you using to mine it and keep it under control?
{ 7 } Comments
Given the fact that Delphi’s compiled programs are behind C# and even Java (!) in terms of execution speed (at least in string/floating point operations), what are compiler guys doing today to make Delphi be as fast as C again?
Alexandre - the compiler teams continue to work on next generation compilers for Delphi and C++Builder. These next generation compilers will start appearing next year and in the years after.
Do you have big data?
yes, we have big data, millions of articles, but how to use them, mine them is a problem.
there are some open source software for this: data mining, text mining. etc.
Hello David I,
I’m currently looking for an data mining solution (it’s getting more and more complex to work with simple SQL Statements / Huge SQL Scripts). I’ll certainly go with Pentaho (open source software).
Hi David,
We have fairly large datasets, measured in multiple gigabytes. One instrument some of our users use records data at about 2GB per minute, although that’s unusual. Our software processes all the data and obviously has to handle this sort of quantity, which isn’t easy in a 32-bit process. We have indexed and caching systems, so we can calculate slices or portions of the end data at a time, and read in from disk only the bits we need. A 64-bit C++ compiler (with good FP code, too!) would be exceptionally handy, mostly because it would allow us to keep more data in memory at any one time.
Cheers,
David
When can I expect Delphi to support GPGPU developments for processing my Big Data?
>Captain said - When can I expect Delphi to support GPGPU developments for processing my Big Data?
We have research projects to do two things with the GPU - 1) graphics/imaging/UI processing, 2) CUDA support. No time frame yet when the research will result with the technologies being included in projects.
Is there specific capabilities you are looking for? CUDA support?
Post a Comment