org.apache.gora.spark

Class GoraSparkEngine<K,V extends Persistent>

java.lang.Object

org.apache.gora.spark.GoraSparkEngine<K,V>

public class GoraSparkEngine<K,V extends Persistent>
extends Object

Base class for Gora - Spark integration.

Constructor Summary

Constructors

Constructor and Description
GoraSparkEngine(Class<K> clazzK, Class<V> clazzV)

Method Summary

Methods

Modifier and Type	Method and Description
<K,V extends Persistent> org.apache.hadoop.conf.Configuration	generateOutputConf(DataStore<K,V> dataStore) Creates a job and sets the output parameters for the conf that Spark will use
<K,V extends Persistent> org.apache.hadoop.conf.Configuration	generateOutputConf(org.apache.hadoop.mapreduce.Job job, Class<? extends DataStore> dataStoreClass, Class<K> keyClass, Class<V> persistentClass) Sets the output parameters for the conf that Spark will use
<K,V extends Persistent> org.apache.hadoop.conf.Configuration	generateOutputConf(org.apache.hadoop.mapreduce.Job job, DataStore<K,V> dataStore, boolean reuseObjects) Sets the output parameters for the conf that Spark will use
org.apache.spark.api.java.JavaPairRDD<K,V>	initialize(org.apache.spark.api.java.JavaSparkContext sparkContext, org.apache.hadoop.conf.Configuration conf, DataStore<K,V> dataStore) Initializes a JavaPairRDD from given Spark context, Hadoop configuration and data store.
org.apache.spark.api.java.JavaPairRDD<K,V>	initialize(org.apache.spark.api.java.JavaSparkContext sparkContext, DataStore<K,V> dataStore) Initializes a JavaPairRDD from given Spark context and data store.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoraSparkEngine

public GoraSparkEngine(Class<K> clazzK,
               Class<V> clazzV)

Method Detail

initialize

public org.apache.spark.api.java.JavaPairRDD<K,V> initialize(org.apache.spark.api.java.JavaSparkContext sparkContext,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    DataStore<K,V> dataStore)

Initializes a JavaPairRDD from given Spark context, Hadoop configuration and data store.

Parameters:

sparkContext - Spark context

conf - Hadoop configuration

dataStore - Data store

Returns:

initialized rdd

initialize

public org.apache.spark.api.java.JavaPairRDD<K,V> initialize(org.apache.spark.api.java.JavaSparkContext sparkContext,
                                                    DataStore<K,V> dataStore)

Initializes a JavaPairRDD from given Spark context and data store. If given data store is Configurable and has not a configuration than a Hadoop configuration is created otherwise existed configuration is used.

Parameters:

sparkContext - Spark context

dataStore - Data store

Returns:

initialized rdd

generateOutputConf

public <K,V extends Persistent> org.apache.hadoop.conf.Configuration generateOutputConf(DataStore<K,V> dataStore)
                                                        throws IOException

Creates a job and sets the output parameters for the conf that Spark will use

Parameters:

dataStore - the datastore as the output

Throws:

IOException

generateOutputConf

public <K,V extends Persistent> org.apache.hadoop.conf.Configuration generateOutputConf(org.apache.hadoop.mapreduce.Job job,
                                                                               DataStore<K,V> dataStore,
                                                                               boolean reuseObjects)

Sets the output parameters for the conf that Spark will use

Parameters:

job - the job to set the properties for

dataStore - the datastore as the output

reuseObjects - whether to reuse objects in serialization

generateOutputConf

public <K,V extends Persistent> org.apache.hadoop.conf.Configuration generateOutputConf(org.apache.hadoop.mapreduce.Job job,
                                                                               Class<? extends DataStore> dataStoreClass,
                                                                               Class<K> keyClass,
                                                                               Class<V> persistentClass)

Sets the output parameters for the conf that Spark will use

Parameters:

job - the job to set the properties for

dataStoreClass - the datastore class

keyClass - output key class

persistentClass - output value class