org.apache.lens.rdd

Class LensRDDClient

java.lang.Object

org.apache.lens.rdd.LensRDDClient

public class LensRDDClient
extends Object

Create RDD from a Lens query. User can poll returned query handle with isReadyForRDD() until the RDD is ready to be used.

Example -

   LensRDDClient client = new LensRDDClient(javaSparkContext);
   QueryHandle query = client.createLensRDDAsync("SELECT msr1 from TEST_CUBE WHERE ...", conf);

   while (!client.isReadyForRDD(query)) {
     Thread.sleep(1000);
   }

   JavaRDD<ResultRow> rdd = client.getRDD(query).toJavaRDD();

   // Consume RDD here -
   rdd.map(...);
 

Alternatively in blocking mode

 JavaRDD<ResultRow> rdd = client.createLensRDD("SELECT msr1 from TEST_CUBE WHERE ...", conf);
 

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	LensRDDClient.LensRDDResult Container object to store the RDD and corresponding Lens query handle.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG The Constant LOG.

Constructor Summary

Constructors

Constructor and Description
LensRDDClient(org.apache.spark.api.java.JavaSparkContext sparkContext) Create an RDD client with given spark Context.
LensRDDClient(org.apache.spark.api.java.JavaSparkContext sparkContext, LensClient lensClient) Instantiates a new lens rdd client.
LensRDDClient(org.apache.spark.SparkContext sc) Create an RDD client with given spark Context.
LensRDDClient(org.apache.spark.SparkContext sparkContext, LensClient lensClient) Instantiates a new lens rdd client.

Method Summary

Methods

Modifier and Type	Method and Description
void	cancelRDD(QueryHandle queryHandle) Allow cancelling underlying query in case of non blocking RDD creation.
LensRDDClient.LensRDDResult	createLensRDD(String query) Blocking call to create an RDD from a Lens query.
QueryHandle	createLensRDDAsync(String query) API for non blocking use.
LensRDDClient.LensRDDResult	getRDD(QueryHandle queryHandle) Get the RDD created for the query.
boolean	isReadyForRDD(QueryHandle queryHandle) Check if the RDD is created.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

The Constant LOG.

Constructor Detail

LensRDDClient

public LensRDDClient(org.apache.spark.api.java.JavaSparkContext sparkContext)

Create an RDD client with given spark Context.

Parameters:

sparkContext - the spark context

LensRDDClient

public LensRDDClient(org.apache.spark.SparkContext sc)

Create an RDD client with given spark Context.

Parameters:

sc - the sc

LensRDDClient

public LensRDDClient(org.apache.spark.api.java.JavaSparkContext sparkContext,
             LensClient lensClient)

Instantiates a new lens rdd client.

Parameters:

sparkContext - the spark context

lensClient - the lens client

LensRDDClient

public LensRDDClient(org.apache.spark.SparkContext sparkContext,
             LensClient lensClient)

Instantiates a new lens rdd client.

Parameters:

sparkContext - the spark context

lensClient - the lens client

Method Detail

createLensRDDAsync

public QueryHandle createLensRDDAsync(String query)
                               throws LensAPIException

API for non blocking use.

Parameters:

query - the query

Returns:

the query handle

Throws:

LensAPIException - the lens exception

isReadyForRDD

public boolean isReadyForRDD(QueryHandle queryHandle)
                      throws LensException

Check if the RDD is created. RDD will be created as soon as the underlying Lens query is complete

Parameters:

queryHandle - the query handle

Returns:

true, if is ready for rdd

Throws:

LensException - the lens exception

cancelRDD

public void cancelRDD(QueryHandle queryHandle)
               throws LensException

Allow cancelling underlying query in case of non blocking RDD creation.

Parameters:

queryHandle - the query handle

Throws:

LensException - the lens exception

getRDD

public LensRDDClient.LensRDDResult getRDD(QueryHandle queryHandle)
                                   throws LensException

Get the RDD created for the query. This should be used only is isReadyForRDD returns true

Parameters:

queryHandle - the query handle

Returns:

the rdd

Throws:

LensException - the lens exception

createLensRDD

public LensRDDClient.LensRDDResult createLensRDD(String query)
                                          throws LensAPIException,
                                                 LensException

Blocking call to create an RDD from a Lens query. Return only when the query is complete.

Parameters:

query - the query

Returns:

the lens rdd result

Throws:

LensException - the lens exception

LensAPIException