Blackfish SQL
We’ve been going about 90 miles an hour this summer working to get the next release of BDS studio out. On the database side of the house there are a ton of new features. I don’t know what we were thinking putting so much into the product and shipping at the end of the summer. The good news is that there’s lots of cool features to talk about now. In this blog I’m going to talk about one of the big features – Blackfish SQL 8.0.
Blackfish SQL is an embeddable database which is now a feature of Delphi. Delphi Pro, Enterprise and Architect customers have unlimited deployment licenses of Blackfish SQL. There are file size and user limits, but these limits should be acceptable for a large number of smaller scale applications. Deployment licenses can be purchased if the user or file size limits need to be increased.
The Blackfish SQL database was written completely in Java for its first 7 revisions. The database product was called JDataStore then. A couple years ago or so Jens Ole created a Java to C# translator so that we could create a .NET version of the database. The bulk of the database kernel, query engine and driver layers are written in Java with minimal dependencies on runtime classes. Since at its core the C# language and .NET runtime is semantically very similar to Java’s, and because we had minimal runtime dependencies, the Java to .NET translator came together fairly quickly (well, much faster than rewriting the database). This translator allows us to continue extending the internals in Java and then automatically translate to C# to produce the .NET edition. There is a very high degree of compatibility between the .NET and Java editions. .NET clients can connect to Java servers and visa versa. The file format is compatible as well.
We also have introduced a 100% Object Pascal DBX4 driver that can connect to Blackfish SQL called DbxClient. Source code for this driver is included in the product. It can be compiled for either win32 or .NET. Because it is 100% Object Pascal and requires no client library you can even build “big fat win32 exes” that connect to either a Blackfish SQL for windows (.NET) or Blackfish SQL for Java server. No additional client library is required. This is a perfect example of why I really like seeing database drivers implemented in Object Pascal. This allows us to use the same code for both .NET and win32 as packages or linked directly into the exe. When there is 64 bit Delphi we can reuse the same code yet again. What other language and runtime can do this as elegantly as Delphi (or even at all)? To me this is one of the many reasons Delphi just flat out rocks.
What else is cool about DbxClient? Well, its not called the Blackfish SQL DBX4 driver. It’s a driver that remotes the interfaces of the DBX4 framework. It can connect to anything that understands its JSON based streaming protocol. Today only the Blackfish SQL for windows and Blackfish SQL for Java support this protocol. However, I’d really like to see if we can leverage this technology as a connectivity solution for DataSnap as well. I view a database driver is a fairly capable RPC layer. The streaming protocol is currently a hybrid where it will mix binary data with text data. In the future we’d also like to implement a full text mode to support other transports like http(s). The binary hybrid mode is more compact and performant.
Another interesting aspect of the DbxClient is the source code. I already mentioned that it was 100% Object Pascal compliable for win32, .NET and someday the win64 platform. If you look at the source code you will notice that some units are located under a pas directory tree and some are located under an xpas directory tree. The source code in the pas directories was authored in Delphi. The source code in the xpas was not. Jens Ole has taught his Java to C# translator a new trick. It now can also translate from Java to Object Pascal. Its pretty hard to tell it came from Java though. Typical Delphi naming conventions are applied as part of the translation process and multiple related classes are packaged into single Delphi units. Often times a Java package of classes logically maps to a Delphi unit. Why do we do this? By using Java as “pseudo code”, we can maintain this JSON based streaming protocol on 3 different platforms: Delphi/C++ win32/win64, .NET and Java. Keep in mind we have a .NET and a Java server at the other end of the DbxClient that need to be able to interpret the same JSON streaming protocol. This same technique was also applied for the new DBX4 metadata.
DBXClient also has some DBX4 extensions. It is the first dbExpress driver we have created that supports streaming blobs and parameter metadata. You don’t need to read the entire contents of the blob in one shot. This is not that big of a deal for things like pictures. But for things like video streams you really want to be able to stream the blob. The DBXClient also supports parameter metadata. If you don’t add parameters to the command, it will add them for you when you prepare the statement. This saves coding and is less error prone than creating parameter objects yourself. I’d like to make these important capabilities part of a “DBX5” in the future.
That’s enough for one blog…
Share This | Email this page to a friend
Posted by Steve Shaughnessy on September 6th, 2007 under Uncategorized |
RSS Feed
September 7th, 2007 at 12:18 am
Guys, instead of wasting time on "3 different platforms" stuff, just buy something really useful, fast, native, with SQL support like AbsoluteDB or similar thing.
I don’t like games CodeGear started to play again, "Ouh, Delphi for NET, amazing! And Ruby is also popular! Do not forget PHP"
Guys, at least make good native IDE. You” be amazed how much more you can do with proper focusing.
September 7th, 2007 at 1:47 am
Can we get/buy Blackfish separately?
Can we use the ‘old’ Delphi 2007 Ent Ed plus the new DBX4 to work with blackfish?
Where does this place Interbase 2007?
Not very clear to me…
September 7th, 2007 at 2:47 am
Very cool stuff, specially parameter Metadata! I’ll be waiting for DBX5 Steve!
If you ask me, the DB team in CodeGear is the only team that still rocks!
But I have a simple question:
Why not use and deploy SQLite, a public domain, absolutely free, open source, community standard, popular, super active, very light, totally tested and robust embeddable database engine? or at least providing a dbexpress driver for it?
I believe if you let people choose, most professionals will go for SQLite rather than Blackfish.
I know you’ve put a lot of time and effort for JDatastore and Blackfish SQL, but I don’t think strategically it will be a very good idea.
Interbase itself is not as popular as it was after huge success of Firebird with it’s embedded library and new (planned) features.
I know it is getting too long for a comment, but I have another request!
Is there a way for Delphi SOAP datasnap applications to send/receive raw XML files instead of binary OLE Variant that ClientDataset sends? This way we can connect to other standard SOA servers.
September 7th, 2007 at 5:49 am
"K A," your comment that "InterBase is not as popular…" post-Firebird is simply wrong. You made that up.
September 7th, 2007 at 7:04 am
Craig, In the circles that my company works, Interbase are not on the cards. All about Firebird. It depends which market segments you look at.
September 7th, 2007 at 8:25 am
Craig,
I don’t have figures for Interbase sales, and do not have figures for Firebird usage, so I can’t support my comment statistically.
But I have one argument:
Despite popular demand in QC, CodeGear is not willing to provide a dbx driver for Firebird. Firbird is the direct/natural competitor for Interbase, so CG is not going to support it.
If Firebird support didn’t have an impact on Interbase sales, CG would think twice for ignoring such market request.
And, I don’t have anything against Interbase. That was my 2 cents!
September 7th, 2007 at 9:56 am
"K A," I don’t presume that you have anything against IB, but you’re still jumping to conclusions without examining data. Setting aside IB, for a moment, which is CodeGear’s own product, and therefore a special case, it’s quite clear to me that every dbx driver CodeGear has written is for a DB server with a *substantially* higher installed base than Firebird. (SQL Server, Oracle, MySQL. The Sybase drivers weren’t written by CodeGear.)
I personally don’t suspect for a moment that CodeGear would hesitate to write a DBX driver for Firebird if they felt that doing so would increase Delphi sales more than spending the same effort on a different Delphi feature.
IB is an established product with a long track record. If the release of Firebird didn’t hurt it, then a DBX driver certainly won’t.
September 7th, 2007 at 10:00 am
The thought of translating Blackfish SQL to Delphi has crossed our mind. However, there are some challenges for thread synchronization and memory management. We also have some higher priority issues to address in the areas of components, DataSnap, and tooling that need to be addressed in the near future.
I have a comments to share on supporting other database products. Although CodeGear has its own database products, the Delphi product continues to be database agnostic. The dbExpress architecture is very open and we provide even more source code for it in this release. I personally have nothing against supporting Firebird or any other database that is important to Delphi customers. However supporting more database backends is a business decision because there is a cost for each one we have to maintain and test. This is one of the reasons we are trying to share as much code as possible across drivers and make it as easy as possible for others to implement dbExpress drivers. I’m not saying we will not add more dbExpress drivers in the future, however I am not the only one involved in this decision.
As for ClientDataSet/DataSnap, this is one of the next big areas we would like to address. I don’t have specifics yet.
Believe it or not, the vast majority of our energy has gone into consolodating and revitalizing our connectivity solutions this past year. It was expensive to maintain two sets of drivers and the win32 customer needed to reap the benefits of any connectivity improvements.
September 7th, 2007 at 10:13 am
Is it currently possible to get at the json output or do we have to wait for you to do some work on your side ?
I’ve noticed in the web space that some client side components take json as input and I was wondering it was possible to feed output into those.
September 7th, 2007 at 10:46 am
The DbxClient JSON streaming layer could probably be used generically for JSON/RPC interactions. However we have not separated this out from the DBX4 driver layer that uses it. The RPCs we create for DBX4 RPCs are too terse and optimized for normal JSON/RPC interactions. We also currently intermix binary data. Going forward it would be good for the server to support typical JSON/RPC interactions. That way we can support more simple clients such as a java script based web client. Note howver that this kind of an interaction requires more redundant data to be sent (ie column names for every value) and less specific metadata for the data values them selves.
September 7th, 2007 at 10:47 am
Steve,
Blackfish sounds great. Keep up the good work.
Who should I contact about making Interbase more attractive to ISPs? I would love to use it fr my web apps, but few ISPs can afford to offer it.
September 7th, 2007 at 2:28 pm
Instead of developing "yet another DB", why not just take the one you have been working on for the last 15+ years and make it embeddable? What’s wrong with making an embedded Interbase? Oh, wait, its already been done by the Firebird developers.
Oh, and when I read that you wrote a Java to C# translator just so the database could be compiled to .net, I laughed out loud (my kids asked what was so funny). You have got to be kidding….
This is a case of creating technology for technology’s sake. It wasn’t needed, isn’t needed, and I predict will be just another footnote in Delphi’s history in a release or two.
Codegear guys, you have limited resources. Put them to use making the base product better. The Help still needs attention as does a lot of stuff in Object Pascal (Unicode, Generics, etc.).
What was not needed was a new DB system.
September 7th, 2007 at 6:32 pm
So, when can we expect to see a Ruby driver?
September 8th, 2007 at 9:54 am
"In the future we’d also like to implement a full text mode to support other transports like http(s)."
HTTP(S) can transport binary data easily - and it does since the beginning - i.e. zlib compressed pages, and HTTP downloads.
The entity-body is defined as *OCTET (any number of OCTETs) and:
OCTET = <any 8-bit sequence of data>
Therefore you don’t need to waste CPU and network time to move around text data.
September 10th, 2007 at 7:58 pm
One of the major pluses to blackfish which I tried to do with firebird(I really like firebird) is putting it on a isp controlled webserver like a lot of us use. Since extra diskspace is a lot cheaper than extra database space on most isp hosted websites this is a major plus.
September 11th, 2007 at 7:15 am
m. Th.: "Lost the battle?" What battle? Firebird, objectively, doesn’t seem to affect IB sales at all, based on actual figures, not blind speculation based on SourceForge’s paper-plate awards. Yes, IB sales went up around the time that Firebird was released, but it makes more sense to credit this to the new management (again, based on having observed both in action, not on some web award somebody got) than anything else.
Like "K A," you’re entirely missing the point: If you want CodeGear to write you a free Firebird driver for DBX, then you need to show them that doing so will increase Delphi sales more than the same effort spent on other features. Misinformed slagging of CodeGear’s other products isn’t going to help your case.
September 11th, 2007 at 11:30 am
As someone who is still maintaining legacy applications that use the BDE, we are still looking for a drop-in replacement for the BDE that doesn’t require hardly any additional overhead. BlackFish SQL could satisfy this niche, but only when it supports TTable, TField, and all of the other BDE constructs we currently use. As for SQL connectivity, we had to write our own SQL layer from scratch because we could not get the dbx and ADO drivers to support all the features that we needed.
September 13th, 2007 at 1:36 am
BlackFish! The name itself suggest that there is something black about this, Not GOOD!
First smooth out your dame IDE. It is pathetic at best. BDS is also very very resource hungry. Better spend some more time and resources on trimming down BDS resource consumption and speed up its loading time.
I don’t know why CG is spending time and resources on developing new things instead of stream lining, fine tuning and speeing up what it has already developed.
I think you people need to learn at least this from M$. Look their Office suite. It was a slot but in Office 2003 version things changed. What we have is a faster better and usable product.
If you want to follow M$ follow it completely in every aspect or don’t follow it at all. Why do things half heartedly.
September 17th, 2007 at 10:24 am
I dont really see how Blackfish can be viewed as Embedded for Win32, If I understand correctly you still need to run the .NET server to connect locally.
Doesnt this kind of defeat the zero configuration ideal? Surely I will still have to deliver .NET runtime to my customers?
Cant say Im thrilled about that.
Whens the "real" native version coming?
September 20th, 2007 at 8:48 am
In my opinion is not CodeGear work to make driver for dbExpres, that’s FireBird community work.
November 4th, 2007 at 6:44 pm
Yep!!! Community must develop that driver for CodeGear… to rise up the sales…
A lot of people is waiting for that damn driver to jump from delphi 7 to delphi 2007… just check on web… (www.google.com) and you all will see that…
Cut the support for Firebird was a wrong market decision…
Firebird is now for Delphi 2007 like Paradox was for Delphi 3… WAKE UP GUYS!!!
Regards!
November 29th, 2007 at 12:58 pm
Please, I need know… What is the advantages that I will be have using BlackFish, comparing with FireBird, InterBase or PostGreSql by example ? Thanks…
January 17th, 2008 at 12:48 pm
We need a committment from CodeGear that DBX4 will support Firebird 2.0.
We use C++Builder and we are evaluating a move to C++Builder 2007.
Not sure how much work it is for CodeGear seeing as they already support Interbase, but adding support to Firebird 2.0 will mean a few purchases of C++Builder 2007 by our company.
April 23rd, 2008 at 10:39 pm
When can we expect to see a "Ruby driver"?
July 16th, 2008 at 2:07 am
Very cool stuff, specially parameter Metadata! I’ll be waiting for DBX5.