Keyv  1.0.0
Key-Value based storage and caching using a variety of backends
keyv::Map Class Reference

Unified interface to save key-value pairs in a store. More...

#include <Map.h>

+ Collaboration diagram for keyv::Map:

Public Member Functions

 Map (const servus::URI &uri)
 Construct a new map. More...
 
 ~Map ()
 Destruct the map. More...
 
size_t setQueueDepth (const size_t depth)
 Set the maximum number of asynchronous outstanding write operations. More...
 
template<class V >
bool insert (const std::string &key, const V &value)
 Insert or update a value in the database. More...
 
bool insert (const std::string &key, const void *data, size_t size)
 
template<class V >
bool insert (const std::string &key, const std::vector< V > &values)
 Insert or update a vector of values in the database. More...
 
template<class V >
bool insert (const std::string &key, const std::set< V > &values)
 Insert or update a set of values in the database. More...
 
std::string operator[] (const std::string &key) const
 Retrieve a value for a key. More...
 
template<class V >
get (const std::string &key) const
 Retrieve a value for a key. More...
 
template<class V >
std::vector< V > getVector (const std::string &key) const
 Retrieve a value as a vector for a key. More...
 
template<class V >
std::set< V > getSet (const std::string &key) const
 Retrieve a value as a set for a key. More...
 
bool fetch (const std::string &key, size_t sizeHint=0) const
 Asynchronously retrieve a value which to be read later. More...
 
void getValues (const Strings &keys, const ConstValueFunc &func) const
 Retrieve values from a list of keys and calls back for each found value. More...
 
void takeValues (const Strings &keys, const ValueFunc &func) const
 Retrieve values from a list of keys and calls back for each found value. More...
 
bool flush ()
 Flush outstanding operations to the backend storage. More...
 
void setByteswap (const bool swap)
 Enable or disable endianness conversion on reads. More...
 
template<>
bool _insert (const std::string &k, const std::string &v, const std::false_type &)
 

Static Public Member Functions

static bool handles (const servus::URI &uri)
 
static MapPtr createCache ()
 Create a map which can be used for caching IO on the local system. More...
 

Detailed Description

Unified interface to save key-value pairs in a store.

Example:

/* Copyright (c) 2014-2016, Stefan.Eilemann@epfl.ch
*
* This file is part of Keyv <https://github.com/BlueBrain/Keyv>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* - Redistributions of source code must retain the above copyright notice, this
* list of conditions and the following disclaimer.
* - Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* - Neither the name of Eyescale Software GmbH nor the names of its
* contributors may be used to endorse or promote products derived from this
* software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
* LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
* CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
* SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
* INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
* CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
* ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
*/
#define TEST_RUNTIME 240 //seconds
#include <keyv/Map.h>
#include <lunchbox/test.h>
#include <lunchbox/clock.h>
#include <lunchbox/os.h>
#include <lunchbox/rng.h>
#ifdef KEYV_USE_LEVELDB
# include <leveldb/db.h>
#endif
#include <boost/format.hpp>
#include <stdexcept>
using keyv::Map;
const int ints[] = { 17, 53, 42, 65535, 32768 };
const size_t numInts = sizeof( ints ) / sizeof( int );
const int64_t loopTime = 1000;
template< class T > void insertVector( Map& map )
{
std::vector< T > vector;
for( size_t i = 0; i < numInts; ++i )
vector.push_back( T( ints[ i ] ));
TEST( map.insert( typeid( vector ).name(), vector ));
}
template< class T > void readVector( const Map& map )
{
const std::vector< T >& vector =
map.getVector< T >( typeid( vector ).name( ));
TESTINFO( vector.size() == numInts, vector.size() << " != " << numInts );
for( size_t i = 0; i < numInts; ++i )
TEST( vector[ i ] == T( ints[i] ));
}
template< class T > void insertVector( Map& map, const size_t elems )
{
std::vector< T > vector;
for( size_t i = 0; i < elems; ++i )
vector.push_back( T(i) );
TEST( map.insert( std::string( "bulk" ) + typeid( vector ).name(), vector ));
}
template< class T > void readVector( const Map& map, const size_t elems )
{
const std::vector< T >& vector =
map.getVector< T >( std::string( "bulk" ) + typeid( vector ).name());
TESTINFO( vector.size() == elems, vector.size() << " != " << elems );
for( size_t i = 0; i < numInts; ++i )
TESTINFO( vector[ i ] == T( i ), vector[ i ] << " != " << i );
}
void read( const Map& map )
{
const std::set< uint32_t >& bigSet =
map.getSet< uint32_t >( "std::set< uint32_t >" );
TEST( bigSet.size() == 1000 );
for( uint32_t i = 1; i <= 1000; ++i )
TEST( bigSet.find( i ) != bigSet.end( ));
TEST( map[ "foo" ] == "bar" );
TEST( map[ "bar" ].empty( ));
TEST( map.get< bool >( "bValue" ) == true );
TEST( map.get< int >( "iValue" ) == 42 );
readVector< int >( map );
readVector< uint16_t >( map );
const std::set< int >& set = map.getSet< int >( "std::set< int >" );
TESTINFO( set.size() == numInts, set.size() << " != " << numInts );
for( size_t i = 0; i < numInts; ++i )
TESTINFO( set.find( ints[i] ) != set.end(),
ints[i] << " not found in set" );
}
void read( const std::string& uriStr )
{
const servus::URI uri( uriStr );
Map map( uri );
read( map );
}
bool testAvailable( const std::string& uriStr )
{
try
{
const servus::URI uri( uriStr );
Map map( uri );
if( !map.insert( "foo", "bar" ))
return false;
return map[ "foo" ] == "bar";
}
catch( ... )
{
return false;
}
}
void setup( const std::string& uriStr )
{
const servus::URI uri( uriStr );
Map map( uri );
TEST( map.insert( "foo", "bar" ));
TESTINFO( map[ "foo" ] == "bar",
map[ "foo" ] << " length " << map[ "foo" ].length( ));
TEST( map[ "bar" ].empty( ));
TEST( map.insert( "the quick brown fox", "jumped over something" ));
TESTINFO( map[ "the quick brown fox" ] == "jumped over something",
map[ "the quick brown fox" ] );
TEST( map.insert( "hans", std::string( "dampf" )));
TESTINFO( map[ "hans" ] == "dampf", map[ "hans" ] );
const bool bValue = true;
TEST( map.insert( "bValue", bValue ));
TEST( map.get< bool >( "bValue" ) == bValue );
const int iValue = 42;
TEST( map.insert( "iValue", iValue ));
TEST( map.get< int >( "iValue" ) == iValue );
TEST( map.insert( "coffee", 0xC0FFEE ));
map.setByteswap( true );
TEST( map.get< unsigned >( "coffee" ) == 0xEEFFC000u );
map.setByteswap( false );
TEST( map.get< int >( "coffee" ) == 0xC0FFEE );
insertVector< int >( map );
insertVector< uint16_t >( map );
readVector< int >( map );
readVector< uint16_t >( map );
insertVector< int >( map, LB_128KB );
insertVector< uint16_t >( map, LB_128KB );
map.fetch( std::string( "bulk" ) + typeid( std::vector< int > ).name( ));
map.fetch( std::string( "bulk" ) + typeid( std::vector<uint16_t> ).name( ));
readVector< int >( map, LB_128KB );
readVector< uint16_t >( map, LB_128KB );
std::set< int > set( ints, ints + numInts );
TEST( map.insert( "std::set< int >", set ));
std::set< uint32_t > bigSet;
for( uint32_t i = 1; i <= 1000; ++i )
bigSet.insert( i );
TEST( map.insert( "std::set< uint32_t >", bigSet ));
const lunchbox::Strings keys = { "hans", "coffee" };
size_t numResults = 0;
map.takeValues( keys, [&]( const std::string& key, char* data,
const size_t size )
{
TEST( std::find( keys.begin(), keys.end(), key) != keys.end( ));
TEST( data );
TESTINFO( size > 0, key << " in " << uriStr );
++numResults;
free( data );
});
TEST( numResults == keys.size( ));
numResults = 0;
map.getValues( keys, [&]( const std::string& key, const char* data,
const size_t size )
{
TEST( std::find( keys.begin(), keys.end(), key) != keys.end( ));
TEST( data );
TEST( size > 0 );
++numResults;
});
TEST( numResults == keys.size( ));
}
void benchmark( const std::string& uriStr, const uint64_t queueDepth,
const size_t valueSize )
{
static std::string lastURI;
if( uriStr != lastURI )
{
std::cout
<< " " << uriStr << std::endl
<< " depth, size, writes/s, MB/s, reads/s, MB/s"
<< std::endl;
lastURI = uriStr;
}
// cppcheck-suppress zerodivcond
std::cout << boost::format( "%6i, %8i,") % queueDepth % valueSize
<< std::flush;
const servus::URI uri( uriStr );
Map map( uri );
map.setQueueDepth( queueDepth );
// Prepare keys and value
lunchbox::Strings keys;
keys.resize( queueDepth + 1 );
for( uint64_t i = 0; i <= queueDepth; ++i )
keys[i].assign( reinterpret_cast< char* >( &i ), 8 );
std::string value( valueSize, '*' );
lunchbox::RNG rng;
for( size_t i = 0; i < valueSize; ++i )
value[i] = rng.get<char>();
// write performance
lunchbox::Clock clock;
uint64_t i = 0;
while( clock.getTime64() < loopTime || i <= queueDepth )
{
map.insert( keys[ i % (queueDepth+1) ], value );
++i;
}
map.flush();
float time = clock.getTimef() / 1000.f;
const uint64_t wOps = i;
// cppcheck-suppress zerodivcond
std::cout << boost::format( "%9.2f, %9.2f,") % (wOps/time)
% (wOps/1024.f/1024.f * valueSize / time) << std::flush;
// read performance
clock.reset();
if( queueDepth == 0 ) // sync read
{
for( i = 0; i < wOps && clock.getTime64() < loopTime; ++i ) // read keys
map[ keys[ i % (queueDepth+1) ]];
}
else // fetch + async read
{
for( i = 0; i < queueDepth; ++i ) // prefetch queueDepth keys
TEST( map.fetch( keys[ i % (queueDepth+1) ], valueSize ) );
for( ; i < wOps && clock.getTime64() < loopTime; ++i ) // read keys
{
map[ keys[ (i - queueDepth) % (queueDepth+1) ] ];
TEST( map.fetch( keys[ i % (queueDepth+1) ], valueSize ));
}
for( uint64_t j = i - queueDepth; j <= i; ++j ) // drain fetched keys
map[ keys[ j % (queueDepth+1) ]];
}
time = clock.getTimef() / 1000.f;
std::cout << boost::format( "%9.2f, %9.2f") % (i/time)
% (i/1024.f/1024.f * valueSize / time) << std::endl;
// try to make sure there's nothing outstanding if we messed up in our test
map.flush();
}
void testGenericFailures()
{
try
{
setup( "foobar://" );
}
catch( const std::runtime_error& )
{
return;
}
TESTINFO( false, "Missing exception" );
}
void testLevelDBFailures()
{
#ifdef KEYV_USE_LEVELDB
try
{
setup( "leveldb:///doesnotexist/deadbeef/coffee" );
}
catch( const std::runtime_error& )
{
return;
}
TESTINFO( false, "Missing exception" );
#endif
}
size_t dup( const size_t value )
{
return value == 0 ? 1 : value << 1;
}
struct TestSpec
{
TestSpec( const std::string& uri_, const size_t depth_, const size_t size_ )
: uri( uri_ ), depth( depth_ ), size( size_ ) {}
std::string uri;
size_t depth;
size_t size;
};
int main( const int argc, char* argv[] )
{
if( argc == 4 )
{
benchmark( argv[1], atoi( argv[2] ), atoi( argv[3] ));
return EXIT_SUCCESS;
}
const bool perfTest =
std::string( argv[0] ).find( "perf-" ) != std::string::npos;
typedef std::vector< TestSpec > TestSpecs;
TestSpecs tests;
#ifdef KEYV_USE_LEVELDB
tests.push_back( TestSpec( "", 0, 65536 ));
tests.push_back( TestSpec( "leveldb://", 0, 65536 ));
tests.push_back( TestSpec( "leveldb://keyvMap2.leveldb", 0, 65536 ));
#endif
#ifdef KEYV_USE_LIBMEMCACHED
if( testAvailable( "memcached://" ))
{
setup( "memcached://" );
read( "memcached://" );
if( perfTest )
for( size_t i=1; i <= 65536; i = i<<2 )
benchmark( "memcached://", 0, i );
}
#endif
#ifdef KEYV_USE_RADOS
tests.push_back( TestSpec(
"ceph://client.vizpoc@vizpoc/home/eilemann/.ceph/ceph.conf",
8192, LB_4MB ));
#endif
try
{
while( !tests.empty( ))
{
const TestSpec test = tests.back();
tests.pop_back();
setup( test.uri );
read( test.uri );
if( perfTest )
{
for( size_t i = 1; i <= test.size; i = i << 2 )
benchmark( test.uri, test.depth, i );
for( size_t i = 0; i <= test.depth; i = dup( i ))
benchmark( test.uri, i, 1024 );
}
}
}
#ifdef KEYV_USE_LEVELDB
catch( const leveldb::Status& status )
{
TESTINFO( !"exception", status.ToString( ));
}
#endif
catch( const std::runtime_error& error )
{
TESTINFO( !"exception", error.what( ));
}
testGenericFailures();
testLevelDBFailures();
return EXIT_SUCCESS;
}

Definition at line 60 of file Map.h.

Constructor & Destructor Documentation

keyv::Map::Map ( const servus::URI &  uri)
explicit

Construct a new map.

Depending on the URI scheme an implementation backend is chosen. If no URI is given, a default one is selected. Available implementations are:

  • ceph://path_to_ceph.conf (if KEYV_USE_RADOS is defined)
  • leveldb://path (if KEYV_USE_LEVELDB is defined)
  • memcached://[server] (if KEYV_USE_LIBMEMCACHED is defined)

If no path is given for leveldb, the implementation uses keyvMap.leveldb in the current working directory.

If no servers are given for memcached, the implementation uses all servers in the MEMCACHED_SERVERS environment variable, or 127.0.0.1. MEMCACHED_SERVERS contains a comma-separated list of servers. Each server contains the address, and optionally a colon-separated port number.

Parameters
urithe storage backend and destination.
Exceptions
std::runtime_errorif no suitable implementation is found.
std::runtime_errorif opening the leveldb failed.
Version
1.9.2
keyv::Map::~Map ( )

Destruct the map.

Version
1.9.2

Member Function Documentation

static MapPtr keyv::Map::createCache ( )
static

Create a map which can be used for caching IO on the local system.

The concrete implementation used depends on the system setup and available backend implementations. If no suitable implementation is found, a null pointer is returned.

The current implementation returns:

  • A memcached-backed cache if libmemcached is available and the environment variable MEMCACHED_SERVERS is set (see constructor documentation for details).
  • A leveldb-backed cache if leveldb is available and LEVELDB_CACHE is set to the path for the leveldb storage.
Returns
a Map for caching IO, or 0.
bool keyv::Map::fetch ( const std::string &  key,
size_t  sizeHint = 0 
) const

Asynchronously retrieve a value which to be read later.

Might be implemented as a 'NOP' by backend implementations.

Parameters
keythe key to retrieve.
sizeHintthe size of the value, may be ignored by implementation.
Returns
false on error, true otherwise.
Version
1.11
bool keyv::Map::flush ( )

Flush outstanding operations to the backend storage.

Version
1.11
template<class V >
V keyv::Map::get ( const std::string &  key) const
inline

Retrieve a value for a key.

Parameters
keythe key to retrieve.
Returns
the value, or an empty string if the key is not available.
Version
1.11

Definition at line 194 of file Map.h.

template<class V >
std::set< V > keyv::Map::getSet ( const std::string &  key) const
inline

Retrieve a value as a set for a key.

Parameters
keythe key to retrieve.
Returns
the values, or an empty set if the key is not available.
Version
1.9.2

Definition at line 350 of file Map.h.

void keyv::Map::getValues ( const Strings &  keys,
const ConstValueFunc func 
) const

Retrieve values from a list of keys and calls back for each found value.

Depending on the backend implementation, this is more optimal than calling get() for each key.

The ownership of the returned data in the callback is not transfered, so the value needs to be copied if needed.

Parameters
keyslist of keys to obtain
funccallback function which is called for each found key
Version
1.14
template<class V >
std::vector< V > keyv::Map::getVector ( const std::string &  key) const
inline

Retrieve a value as a vector for a key.

Parameters
keythe key to retrieve.
Returns
the values, or an empty vector if the key is not available.
Version
1.9.2

Definition at line 336 of file Map.h.

static bool keyv::Map::handles ( const servus::URI &  uri)
static
Returns
true if an implementation for the given URI is available.
Version
1.9.2
template<class V >
bool keyv::Map::insert ( const std::string &  key,
const V &  value 
)
inline

Insert or update a value in the database.

Parameters
keythe key to store the value.
valuethe value stored at the key.
Returns
true on success, false otherwise
Exceptions
std::runtime_errorif the value is not copyable
Version
1.9.2

Definition at line 139 of file Map.h.

template<class V >
bool keyv::Map::insert ( const std::string &  key,
const std::vector< V > &  values 
)
inline

Insert or update a vector of values in the database.

Parameters
keythe key to store the value.
valuesthe values stored at the key.
Returns
true on success, false otherwise
Exceptions
std::runtime_errorif the vector values are not copyable
Version
1.9.2

Definition at line 158 of file Map.h.

template<class V >
bool keyv::Map::insert ( const std::string &  key,
const std::set< V > &  values 
)
inline

Insert or update a set of values in the database.

Parameters
keythe key to store the value.
valuesthe values stored at the key.
Returns
true on success, false otherwise
Exceptions
std::runtime_errorif the set values are not copyable
Version
1.9.2

Definition at line 175 of file Map.h.

std::string keyv::Map::operator[] ( const std::string &  key) const

Retrieve a value for a key.

Parameters
keythe key to retrieve.
Returns
the value, or an empty string if the key is not available.
Version
1.9.2
void keyv::Map::setByteswap ( const bool  swap)

Enable or disable endianness conversion on reads.

Version
1.11
size_t keyv::Map::setQueueDepth ( const size_t  depth)

Set the maximum number of asynchronous outstanding write operations.

Some backend implementations support asynchronous writes, which can be enabled by setting a non-zero queue depth. Applications then need to quarantee that the inserted values stay valid until 'depth' other elements have been inserted or flush() has been called. Implementations which do not support asynchronous writes return 0.

Returns
the queue depth chosen by the implementation, smaller or equal to the given depth.
Version
1.11
void keyv::Map::takeValues ( const Strings &  keys,
const ValueFunc func 
) const

Retrieve values from a list of keys and calls back for each found value.

Depending on the backend implementation, this is more optimal than calling get() for each key.

The ownership of the returned data in the callback is transfered, so the data must be free'd by the caller.

Parameters
keyslist of keys to obtain
funccallback function which is called for each found key
Version
1.14

The documentation for this class was generated from the following file: