This page is for references to signed quotes of deployments of large triples stores rather than predictions of what some software might scale to.

YARS2 (7B)

YARS2: A Federated Repository for Querying Graph Structured Data from the Web describes the distributed architecture of the YARS2 quad store. With scalability experiments up to 7bn synthetically generated statements - LUBM(50000).

Proprietary

Garlik JXT (1.7B)

"1.25 billion triples worth, at the moment" -- Steve Harris, Garlik

The store is called JXT. Currently we have 4 KBs of 1.6-1.7GT each loaded in our production systems. Loading time for one 1.7GT KB is about 8 hours, but it's an interactive process that involves running lots of queries, and doing small inserts.

In testing we've gone to 3.2GT, but we've not used that much in our production environment. It seemed fine though.

Proprietary

BigOWLIM (1.06B explicit, 1.85B total)

"BigOWLIM successfully passed the threshold of 1 billion (10^9) statements of OWL/RDF - it loaded an 8000-university dataset of the LUBM benchmark and answered the evaluation queries correctly

... Loading, inference, and storage took 69 hours and 51 min; LUBM(8000,0) contains 1.06 billions of explicit statements; The "inferred closure" contains about 786M statements; BigOWLIM had to manage over 1.85 billions of statements in total; 92GB RDF/XML files; 95 GB binary storage files; Average Speed: 4 538 statements/sec." -- Damyan Ognyanoff, Ontotext

Commercial, free limited version on http://www.ontotext.com/owlim/index.html

http://www.ontotext.com/owlim/big/index.html

Virtuoso Open-Source Edition (1B+)

New Bitmap Indexing white paper shows how OpenLink Virtuoso handles loading the 1 billion triple LUBM benchmark set with a sustained rate of 12692 triples/s and the 47M triple Wikipedia data set at a rate of 20800 triples/s. Kingsley Idehen, OpenLink Software.

"The single query stream rate with 100K triples is 14 qps at 100K triples and 11 qps at 1G triples" -- LUBM and Virtuoso

Open source.

http://virtuoso.openlinksw.com/wiki/main/

AllegroGraph (1B)

"This is with version 1.2.4. Our performance numbers are published on our website." -- Steve Sears

Commercial, limited free version.

http://agraph.franz.com/allegrograph/

Jena SDB (650M)

SDB is a new SPARQL database for graphs/named graphs for Jena. Can load UniProt (650M). Uses PostgreSQL, MySQL, Oracle or MS SQL Server. Also, HSQLDB and Apache Derby.

Open source

http://jena.sourceforge.net/SDB/

Mulgara (500M)

"The Mulgara triple store is scalable up to 0.5billion triples (with 64-bit Java)" -- Norman Gray

Open source

http://www.mulgara.org/

RDF gateway (262M)

"the UniProt protein database (262 million triples) and RDF Gateway." -- Geoff Chappell, Intellidimension

Commercial

http://www.intellidimension.com/

Jena with PostgreSQL (200M)

"Our store is pretty big -- its about 200M triples.

We're currently using Jena on Postgres. For our needs this worked out better than Jena/MySQL, Sesame, and Kowari." -- Leigh Dodds, Ingenta

Open source

http://jena.sourceforge.net/

Kowari (160M)

"My own testing has been in the 10-20M triple range." -- Chris Wilper

Addendum from Chris on Nov 7th, 2005: Since this was written, we have successfully loaded over 160M triples into Kowari on a 64-bit machine with 6GB physical memory. A 64-bit machine is really required to bring Kowari up to this level because it uses mapped files and needs a lot of address space. In our experience in this environment, simple queries still perform fairly well (a few seconds) and complex queries involving 8-10 triple patterns perform worse (a few minutes to an hour).

Open source, unmaintained (See Mulgara fork).

http://www.kowari.org/

3store with MySQL 3 (100M)

"The store my consortium produces (3store) is used successfully up to 100M triples or so. Beyond that it gets a bit sketchy. I'm currently looking at ways to make it scale to 10^9+ without specialising the store to a particular schema."

More specifically, one user is running it with 120M triples in MySQL 4.1. At that size query works fine, but assertion time is down to about 300 triples/second, which makes growing it any bigger too painful. I should note that 3store is an RDFS store, in version 3 it's possible to disable the inference, which should make it scale to much larger sizes, but there are plenty of other stores that can run vanilla RDF storage well. -- Steve Harris, AKT

Open source (GNU GPL).

http://threestore.sourceforge.net/

Sesame (70M)

(10-20 million triples) " is a lot, but most serious triple stores can handle this I'd say. Sesame certainly can, ..." -- Jeen Broekstra, Aduna

Addendum from Jeen on Feb 10 2006: The above comment should be taken as a minimum of what the store can handle. We recently have ran a few scalability tests on Sesame's Native Store (Sesame 2.0-alpha-3). Using the Lehigh University Benchmark we successfully added a LUBM-500 dataset (consisting of about 70 million RDF triples). The machine used was a 2.8GhZ P4 (32-bits) with 1GB RAM, running Suse Linux 10.0 (kernel 2.6), Sun J2SE 1.5.0_06. Upload took about 3 hours. Query performance on the LUBM test-queries was adequate to good: unoptimized, the worst query (Q2) took 1.3 hours to complete, but most queries completed within tens of milliseconds (Q4,5,6,7,8,10,12,13) or 1-5 minutes (Q1,3,9,11,14) - though some of these queries are just fast because they return no results (the native store does no RDFS/OWL inferencing). We have yet to explore larger datasets and performance using RDFS inferencing but it seems that 70M is not the ceiling and that Sesame can easily cope with even larger sets, especially when we use bigger hardware. But that's prediction not fact so I'll leave it at that for now ;)

Open source.

http://www.openrdf.org/

Others who claim to go big

Claims without signatures or quotes. Please move them from this section when they can be linked to a signed specific capacity measurement.

Questions

I know storing 200M triples is cool. But which store can handle simultaneous queries of about 10.000 users using RDFS inferencing? -- Anonymous

200M-300M or so seems to be about the max that anybody has reported. It would be very helpful if people could state whether they tried to scale further, and if not able to, what the problems were -- i.e., does it become too slow to add, perform trivial queries, perform complex queries, all of the above, etc. Additionally, it would be extremely helpful if hardware specs were included. Anyway, this is a great resource. -- CS

It would be nice of the postings here comment on the level of inference supported. Loading with forward-chaining and materialization is *much* heavier than just loading the data. The more general question is what part of the semantics of the loaded ontology/dataset is supported by the system. There are subtle differences in what "loading" means for the four systems with highest results above. RDF gateway supports the semantics of UNIPROT through backward-chaining. OWLIM supports the semantics of LUBM through forward-chaining. The sort of reasoning required in the UNIPROT load of RDF Gateway is much more complex than the one necessary for passing LUBM. Finally, Virtuoso and AllegroGraph are fairly undetermined with respect to reasoning involved in the experiments they report on. For instance, Virtuoso reports results on LUBM but says nothing about the completeness of the query evaluation. -- Atanas Kiryakov

Related


CategorySwTools

LargeTripleStores (last edited 2008-04-28 09:55:48 by Nicolas Raoul)