org.apache.lens.cube.metadata

Class CubeMetastoreClient

java.lang.Object

org.apache.lens.cube.metadata.CubeMetastoreClient

public class CubeMetastoreClient
extends Object

Wrapper class around Hive metastore to do cube metastore operations.

Method Summary

Modifier and Type	Method and Description
static Map<String,String>	addFactColStartTimePropertyToFactProperties(XFactTable fact)
List<org.apache.hadoop.hive.ql.metadata.Partition>	addPartition(StoragePartitionDesc partSpec, String storageName, CubeTableType type) Add a partition specified by the storage partition desc on the storage passed.
List<org.apache.hadoop.hive.ql.metadata.Partition>	addPartitions(List<StoragePartitionDesc> storagePartitionDescs, String storageName, CubeTableType type) batch addition
void	addStorage(CubeDimensionTable dim, String storage, UpdatePeriod dumpPeriod, StorageTableDesc storageTableDesc) Adds storage to dimension and creates corresponding storage table
void	addStorage(CubeFactTable fact, String storage, Set<UpdatePeriod> updatePeriods, Map<String,StorageTableDesc> storageTableDescs, Map<UpdatePeriod,String> updatePeriodStoragePrefix) Adds storage to fact and creates corresponding storage table
void	alterCube(String cubeName, CubeInterface cube) Alter cube specified by the name to new definition
void	alterCube(XCube cube)
void	alterCubeDimensionTable(String dimTableName, CubeDimensionTable cubeDimensionTable, Map<String,StorageTableDesc> storageTableDescs) Alter dimension table with new dimension definition and underlying storage tables as well
void	alterCubeDimensionTable(XDimensionTable dimensionTable)
void	alterCubeFactTable(String factTableName, FactTable cubeFactTable, Map<String,StorageTableDesc> storageTableDescs, Map<String,String> props) Alter a cubefact with new definition and alter underlying storage tables as well.
void	alterCubeFactTable(XFact fact)
void	alterDimension(String dimName, Dimension newDim)
void	alterDimension(XDimension newDim) Alter dimension specified by the dimension name to new definition
void	alterHiveTable(String table, org.apache.hadoop.hive.ql.metadata.Table hiveTable)
void	alterSegmentation(String segName, Segmentation seg)
void	alterSegmentation(XSegmentation cubeSeg)
void	alterStorage(String storageName, Storage storage)
void	alterStorage(XStorage storage) Alter storage specified by the name to new definition
void	alterVirtualCubeFactTable(CubeVirtualFactTable cubeVirtualFactTable) Alter a virtual cube fact with new definition
void	clearHiveTableCache() clear hive table cache
static void	close() Close the current metastore client
void	createCube(CubeInterface cube) Create cube in metastore defined by Cube or DerivedCube object
void	createCube(String name, Set<CubeMeasure> measures, Set<CubeDimAttribute> dimensions) Create cube defined by measures and dimensions
void	createCube(String name, Set<CubeMeasure> measures, Set<CubeDimAttribute> dimensions, Map<String,String> properties) Create cube defined by measures, dimensions and properties
void	createCube(String name, Set<CubeMeasure> measures, Set<CubeDimAttribute> dimensions, Set<ExprColumn> expressions, Map<String,String> properties) Create cube defined by measures, dimensions and properties
void	createCube(String name, Set<CubeMeasure> measures, Set<CubeDimAttribute> dimensions, Set<ExprColumn> expressions, Set<JoinChain> chains, Map<String,String> properties) Create cube defined by measures, dimensions and properties
void	createCube(XCube cube)
void	createCubeDimensionTable(String dimName, String dimTblName, List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns, double weight, Map<String,UpdatePeriod> dumpPeriods, Map<String,String> properties, Map<String,StorageTableDesc> storageTableDescs) Create a cube dimension table
void	createCubeDimensionTable(String dimName, String dimTblName, List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns, double weight, Set<String> storageNames, Map<String,String> properties, Map<String,StorageTableDesc> storageTableDescs) Create a cube dimension table
void	createCubeDimensionTable(XDimensionTable xDimTable)
void	createCubeFactTable(String cubeName, String factName, List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns, Map<String,Set<UpdatePeriod>> storageAggregatePeriods, double weight, Map<String,String> properties, Map<String,StorageTableDesc> storageTableDescs) Create a cube fact table
void	createCubeFactTable(String cubeName, String factName, List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns, Map<String,Set<UpdatePeriod>> storageAggregatePeriods, double weight, Map<String,String> properties, Map<String,StorageTableDesc> storageTableDescs, Map<String,Map<UpdatePeriod,String>> storageUpdatePeriodMap)
void	createCubeTable(AbstractCubeTable cubeTable, Map<String,StorageTableDesc> storageTableDescs) Create cube table defined and create all the corresponding storage tables
void	createDerivedCube(String parent, String name, Set<String> measures, Set<String> dimensions, Map<String,String> properties, double weight) Create derived cube defined by measures, dimensions and properties
void	createDimension(Dimension dim) Create dimension in metastore defined by Dimension object
void	createDimension(String name, Set<CubeDimAttribute> attributes, Map<String,String> properties, double weight) Create dimension defined by attributes and properties
void	createDimension(XDimension dim)
<T extends org.jvnet.jaxb2_commons.lang.Equals & org.jvnet.jaxb2_commons.lang.HashCode & org.jvnet.jaxb2_commons.lang.ToString> void	createEntity(T entity)
void	createFactTable(XFact fact)
void	createSegmentation(Segmentation cubeSeg)
void	createSegmentation(String baseCubeName, String segmentationName, Set<Segment> segments, double weight, Map<String,String> properties)
void	createSegmentation(XSegmentation cubeSeg)
void	createStorage(Storage storage)
void	createStorage(XStorage storage)
void	createVirtualFactTable(String cubeName, String virtualFactName, String sourceFactName, Double weight, Map<String,String> properties)
boolean	dimTableLatestPartitionExists(String storageTableName)
void	dropCube(String cubeName) Drop a cube
void	dropDimension(String dimName) Drop a dimension
void	dropDimensionTable(String dimTblName, boolean cascade) Drop the dimension table
void	dropFact(String factName, boolean cascade) Drop a fact with cascade flag
void	dropHiveTable(String table)
void	dropPartition(String cubeTableName, String storageName, Map<String,Date> timePartSpec, Map<String,String> nonTimePartSpec, UpdatePeriod updatePeriod) Add a partition specified by the storage partition desc on the storage passed.
void	dropSegmentation(String segName)
void	dropStorage(String storageName) Drop a storage
void	dropStorageFromDim(String dimTblName, String storage) Drop a storage from dimension
void	dropStorageFromFact(String factName, String storage) Drop a storage from fact
void	dropVirtualFact(String virtualFactName) Drop a virtual fact
boolean	factPartitionExists(FactTable fact, FactPartition part, String storageTableName) extract storage name and check in timeline cache for existance
boolean	factPartitionExists(String factName, String storageName, UpdatePeriod updatePeriod, Map<String,Date> partitionTimestamp, Map<String,String> partSpec)
Collection<CubeInterface>	getAllCubes() Get all cubes in metastore
List<DerivedCube>	getAllDerivedQueryableCubes(CubeInterface cube) Get all derived cubes of the cube, that have all fields queryable together
Collection<Dimension>	getAllDimensions() Get all cubes in metastore
Collection<CubeDimensionTable>	getAllDimensionTables() Get all dimension tables in metastore
List<CubeDimensionTable>	getAllDimensionTables(Dimension dim) Get all dimension tables of the dimension.
Collection<FactTable>	getAllFacts() Get all facts in metastore
Collection<FactTable>	getAllFacts(boolean includeVirtualFacts) Get all facts in metastore (virtual facts optional)
List<FactTable>	getAllFacts(CubeInterface cube) Get all fact tables of the cube.
List<FactTable>	getAllFacts(CubeInterface cube, boolean includeVirtualFacts) Get all facts of cube (optional virtual facts)
List<org.apache.hadoop.hive.ql.metadata.Partition>	getAllParts(String storageTableName)
Collection<Segmentation>	getAllSegmentations() Get all segmentations in metastore
List<Segmentation>	getAllSegmentations(CubeInterface cube)
Collection<Storage>	getAllStorages() Get all storages in metastore
DataCompletenessChecker	getCompletenessChecker()
org.apache.hadoop.hive.conf.HiveConf	getConf() Get cube metastore client conf
CubeInterface	getCube(String tableName) Get Cube object corresponding to the name
CubeFactTable	getCubeFactTable(String factName)
CubeVirtualFactTable	getCubeVirtualFactTable(String factName)
Dimension	getDimension(String tableName) Get Cube object corresponding to the name
CubeDimensionTable	getDimensionTable(String tableName) Get CubeDimensionTable object corresponding to the name
FactTable	getFactTable(String tableName) Get Cube object corresponding to the name
org.apache.hadoop.hive.ql.metadata.Table	getHiveTable(String tableName) Get the hive Table corresponding to the name
static CubeMetastoreClient	getInstance(org.apache.hadoop.hive.conf.HiveConf conf) Get the instance of CubeMetastoreClient corresponding to HiveConf
Date	getLatestDateOfCube(Cube cube, String timeDimension) get latest date for timeDimension from all fact-storage tables belonging to the given cube having timeDimension, return the most recent of all.
org.apache.hadoop.hive.ql.metadata.Partition	getPartitionByFilter(String storageTableName, String filter)
List<org.apache.hadoop.hive.ql.metadata.Partition>	getPartitionsByFilter(String storageTableName, String filter)
Segmentation	getSegmentation(String segName)
Segmentation	getSegmentation(String segName, boolean throwException)
Segmentation	getSegmentationTable(String tableName)
Storage	getStorage(String storageName) Get Storage object corresponding to the name
Storage	getStorage(String storageName, boolean throwException)
Date	getStorageTableEndDate(String storageTable, FactTable factTableName)
String	getStorageTableName(String factOrDimTableName, String storage, UpdatePeriod updatePeriod)
Set<String>	getStorageTables(FactTable factTable, String storage) Returns storage table names for a storage.
Date	getStorageTableStartDate(String storageTable, FactTable factTableName)
org.apache.hadoop.hive.ql.metadata.Table	getTable(String tableName)
org.apache.hadoop.hive.ql.metadata.Table	getTable(String tableName, boolean throwException)
org.apache.hadoop.hive.ql.metadata.Table	getTableWithType(String tableName, CubeTableType type, boolean throwException)
org.apache.hadoop.hive.ql.metadata.Table	getTableWithTypeFailFast(String tableName, CubeTableType type)
List<PartitionTimeline>	getTimelines(String factName, String storage, String updatePeriodStr, String timeDimension)
List<String>	getTimePartColNamesOfTable(String storageTableName)
List<String>	getTimePartColNamesOfTable(org.apache.hadoop.hive.ql.metadata.Table table) extract from table properties
XDimensionTable	getXDimensionTable(CubeDimensionTable dimTable)
XDimensionTable	getXDimensionTable(String dimTable)
XFact	getXFactTable(FactTable ft)
XFact	getXFactTable(String tableName)
boolean	isCube(String tableName) Is the table name passed a cube?
boolean	isDataCompletenessCheckEnabled()
boolean	isDimension(String tableName) Is the table name passed a dimension?
boolean	isDimensionTable(String tableName) Is the table name passed a dimension table?
boolean	isFactTable(String tableName) Is the table name passed a fact table?
boolean	isLensQueryableTable(String tableName)
boolean	isStorageTableCandidateForRange(String storageTableName, Date fromDate, Date toDate)
boolean	isStorageTableCandidateForRange(String storageTableName, String fromDate, String toDate)
boolean	isStorageTablePartitionACandidate(String storageTableName, Date partDate)
boolean	isVirtualFactTable(String virtualTableName) Is the table name passed a virtual fact table?
boolean	partColExists(FactTable factTable, String storage, String partCol)
boolean	partitionExists(String storageTableName, UpdatePeriod updatePeriod, Map<String,Date> partitionTimestamps)
boolean	partitionExistsByFilter(String storageTableName, String filter)
boolean	partitionExistsByFilter(String cubeTableName, String storageName, UpdatePeriod updatePeriod, String filter)
void	pushHiveTable(org.apache.hadoop.hive.ql.metadata.Table hiveTable)
boolean	tableExists(String tblName)
<T extends org.jvnet.jaxb2_commons.lang.Equals & org.jvnet.jaxb2_commons.lang.HashCode & org.jvnet.jaxb2_commons.lang.ToString> void	updateEntity(String name, T entity)
void	updatePartition(String fact, String storageName, org.apache.hadoop.hive.ql.metadata.Partition partition, UpdatePeriod updatePeriod)
void	updatePartitions(String factOrDimtableName, String storageName, Map<UpdatePeriod,List<org.apache.hadoop.hive.ql.metadata.Partition>> partitions)
void	verifyStorageExists(AbstractCubeTable cdt, String storage)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getCompletenessChecker

public DataCompletenessChecker getCompletenessChecker()

isDataCompletenessCheckEnabled

public boolean isDataCompletenessCheckEnabled()

getLatestDateOfCube

public Date getLatestDateOfCube(Cube cube,
                                String timeDimension)
                         throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                LensException

get latest date for timeDimension from all fact-storage tables belonging to the given cube having timeDimension, return the most recent of all.

latest date for a single fact-storage table for given time dimension is the latest of the latest dates for all its update periods

Parameters:

cube - Cube to get latest date of

timeDimension - time dimension

Returns:

latest date among all facts of cube in timeDimension

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

clearHiveTableCache

public void clearHiveTableCache()

clear hive table cache

getTimelines

public List<PartitionTimeline> getTimelines(String factName,
                                            String storage,
                                            String updatePeriodStr,
                                            String timeDimension)
                                     throws LensException,
                                            org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

updatePartition

public void updatePartition(String fact,
                            String storageName,
                            org.apache.hadoop.hive.ql.metadata.Partition partition,
                            UpdatePeriod updatePeriod)
                     throws org.apache.hadoop.hive.ql.metadata.HiveException,
                            org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                            LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

org.apache.hadoop.hive.metastore.api.InvalidOperationException

LensException

updatePartitions

public void updatePartitions(String factOrDimtableName,
                             String storageName,
                             Map<UpdatePeriod,List<org.apache.hadoop.hive.ql.metadata.Partition>> partitions)
                      throws org.apache.hadoop.hive.ql.metadata.HiveException,
                             org.apache.hadoop.hive.metastore.api.InvalidOperationException,
                             LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

org.apache.hadoop.hive.metastore.api.InvalidOperationException

LensException

isLensQueryableTable

public boolean isLensQueryableTable(String tableName)

verifyStorageExists

public void verifyStorageExists(AbstractCubeTable cdt,
                                String storage)
                         throws LensException

Throws:

LensException

createCubeFactTable

public void createCubeFactTable(String cubeName,
                                String factName,
                                List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns,
                                Map<String,Set<UpdatePeriod>> storageAggregatePeriods,
                                double weight,
                                Map<String,String> properties,
                                Map<String,StorageTableDesc> storageTableDescs,
                                Map<String,Map<UpdatePeriod,String>> storageUpdatePeriodMap)
                         throws LensException

Throws:

LensException

createEntity

public <T extends org.jvnet.jaxb2_commons.lang.Equals & org.jvnet.jaxb2_commons.lang.HashCode & org.jvnet.jaxb2_commons.lang.ToString> void createEntity(T entity)
                                                                                                                                                  throws LensException

Throws:

LensException

updateEntity

public <T extends org.jvnet.jaxb2_commons.lang.Equals & org.jvnet.jaxb2_commons.lang.HashCode & org.jvnet.jaxb2_commons.lang.ToString> void updateEntity(String name,
                                                                                                                                                         T entity)
                                                                                                                                                  throws LensException,
                                                                                                                                                         org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

addFactColStartTimePropertyToFactProperties

public static Map<String,String> addFactColStartTimePropertyToFactProperties(XFactTable fact)

createFactTable

public void createFactTable(XFact fact)
                     throws LensException

Throws:

LensException

createVirtualFactTable

public void createVirtualFactTable(String cubeName,
                                   String virtualFactName,
                                   String sourceFactName,
                                   Double weight,
                                   Map<String,String> properties)
                            throws LensException

Throws:

LensException

getInstance

public static CubeMetastoreClient getInstance(org.apache.hadoop.hive.conf.HiveConf conf)
                                       throws org.apache.hadoop.hive.ql.metadata.HiveException

Get the instance of CubeMetastoreClient corresponding to HiveConf

Parameters:

conf - conf

Returns:

CubeMetastoreClient instance

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

getConf

public org.apache.hadoop.hive.conf.HiveConf getConf()

Get cube metastore client conf

Returns:

HiveConf

close

public static void close()

Close the current metastore client

createStorage

public void createStorage(XStorage storage)
                   throws LensException

Throws:

LensException

createStorage

public void createStorage(Storage storage)
                   throws LensException

Throws:

LensException

createCube

public void createCube(XCube cube)
                throws LensException

Throws:

LensException

createCube

public void createCube(CubeInterface cube)
                throws LensException

Create cube in metastore defined by Cube or DerivedCube object

Parameters:

cube - the Cube object.

Throws:

LensException

createCube

public void createCube(String name,
                       Set<CubeMeasure> measures,
                       Set<CubeDimAttribute> dimensions)
                throws LensException

Create cube defined by measures and dimensions

Parameters:

name - Name of the cube

measures - Measures of the cube

dimensions - Dimensions of the cube

Throws:

LensException

createCube

public void createCube(String name,
                       Set<CubeMeasure> measures,
                       Set<CubeDimAttribute> dimensions,
                       Map<String,String> properties)
                throws LensException

Create cube defined by measures, dimensions and properties

Parameters:

name - Name of the cube

measures - Measures of the cube

dimensions - Dimensions of the cube

properties - Properties of the cube

Throws:

LensException

createCube

public void createCube(String name,
                       Set<CubeMeasure> measures,
                       Set<CubeDimAttribute> dimensions,
                       Set<ExprColumn> expressions,
                       Map<String,String> properties)
                throws LensException

Create cube defined by measures, dimensions and properties

Parameters:

name - Name of the cube

measures - Measures of the cube

dimensions - Dimensions of the cube

properties - Properties of the cube

Throws:

LensException

createCube

public void createCube(String name,
                       Set<CubeMeasure> measures,
                       Set<CubeDimAttribute> dimensions,
                       Set<ExprColumn> expressions,
                       Set<JoinChain> chains,
                       Map<String,String> properties)
                throws LensException

Create cube defined by measures, dimensions and properties

Parameters:

name - Name of the cube

measures - Measures of the cube

dimensions - Dimensions of the cube

expressions - Expressions of the cube

chains - JoinChains of the cube

properties - Properties of the cube

Throws:

LensException

createDimension

public void createDimension(String name,
                            Set<CubeDimAttribute> attributes,
                            Map<String,String> properties,
                            double weight)
                     throws LensException

Create dimension defined by attributes and properties

Parameters:

name - Name of the dimension

attributes - Attributes of the dimension

properties - Properties of the dimension

weight - Weight of the dimension

Throws:

LensException

createDimension

public void createDimension(XDimension dim)
                     throws LensException

Throws:

LensException

createDimension

public void createDimension(Dimension dim)
                     throws LensException

Create dimension in metastore defined by Dimension object

Parameters:

dim - the Dimension object.

Throws:

LensException

createDerivedCube

public void createDerivedCube(String parent,
                              String name,
                              Set<String> measures,
                              Set<String> dimensions,
                              Map<String,String> properties,
                              double weight)
                       throws LensException

Create derived cube defined by measures, dimensions and properties

Parameters:

parent - Name of the parent cube

name - Name of the derived cube

measures - Measures of the derived cube

dimensions - Dimensions of the derived cube

properties - Properties of the derived cube

weight - Weight of the derived cube

Throws:

LensException

createCubeFactTable

public void createCubeFactTable(String cubeName,
                                String factName,
                                List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns,
                                Map<String,Set<UpdatePeriod>> storageAggregatePeriods,
                                double weight,
                                Map<String,String> properties,
                                Map<String,StorageTableDesc> storageTableDescs)
                         throws LensException

Create a cube fact table

Parameters:

cubeName - The cube name to which fact belongs to.

factName - The fact name

columns - The columns of fact table

storageAggregatePeriods - Aggregate periods for the storages

weight - Weight of the cube

properties - Properties of fact table

storageTableDescs - Map of storage table prefix to its storage table description

Throws:

LensException

createSegmentation

public void createSegmentation(String baseCubeName,
                               String segmentationName,
                               Set<Segment> segments,
                               double weight,
                               Map<String,String> properties)
                        throws LensException

Parameters:

baseCubeName - The cube name ot which segmentation belong to

segmentationName - The segmentation name

segments - Participating cube segements

weight - Weight of segmentation

properties - Properties of segmentation

Throws:

LensException

createCubeDimensionTable

public void createCubeDimensionTable(XDimensionTable xDimTable)
                              throws LensException

Throws:

LensException

createCubeDimensionTable

public void createCubeDimensionTable(String dimName,
                                     String dimTblName,
                                     List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns,
                                     double weight,
                                     Set<String> storageNames,
                                     Map<String,String> properties,
                                     Map<String,StorageTableDesc> storageTableDescs)
                              throws LensException

Create a cube dimension table

Parameters:

dimName - The dimension name to which the dim-table belongs to

dimTblName - dimension table name

columns - Columns of the dimension table

weight - Weight of the dimension table

storageNames - Storages on which dimension is available without any dumps

properties - Properties of dimension table

storageTableDescs - Map of storage to its storage table description

Throws:

LensException

createCubeDimensionTable

public void createCubeDimensionTable(String dimName,
                                     String dimTblName,
                                     List<org.apache.hadoop.hive.metastore.api.FieldSchema> columns,
                                     double weight,
                                     Map<String,UpdatePeriod> dumpPeriods,
                                     Map<String,String> properties,
                                     Map<String,StorageTableDesc> storageTableDescs)
                              throws LensException

Create a cube dimension table

Parameters:

dimName - The dimension name to which the dim-table belongs to

dimTblName - dimension table name

columns - Columns of the dimension table

weight - Weight of the dimension table

dumpPeriods - Storage names and their dump periods on which dimension is available

properties - properties of dimension table

storageTableDescs - Map of storage to its storage table description

Throws:

LensException

createCubeTable

public void createCubeTable(AbstractCubeTable cubeTable,
                            Map<String,StorageTableDesc> storageTableDescs)
                     throws LensException

Create cube table defined and create all the corresponding storage tables

Parameters:

cubeTable - Can be fact or dimension table

storageTableDescs - Map of storage tableName prefix to its storage table description

Throws:

LensException

createSegmentation

public void createSegmentation(XSegmentation cubeSeg)
                        throws LensException

Throws:

LensException

createSegmentation

public void createSegmentation(Segmentation cubeSeg)
                        throws LensException

Throws:

LensException

addStorage

public void addStorage(CubeFactTable fact,
                       String storage,
                       Set<UpdatePeriod> updatePeriods,
                       Map<String,StorageTableDesc> storageTableDescs,
                       Map<UpdatePeriod,String> updatePeriodStoragePrefix)
                throws LensException

Adds storage to fact and creates corresponding storage table

Parameters:

fact - The CubeFactTable

storage - The storage

updatePeriods - Update periods of the fact on the storage

storageTableDescs - The storage table description

Throws:

LensException

addStorage

public void addStorage(CubeDimensionTable dim,
                       String storage,
                       UpdatePeriod dumpPeriod,
                       StorageTableDesc storageTableDesc)
                throws LensException

Adds storage to dimension and creates corresponding storage table

Parameters:

dim - The CubeDimensionTable

storage - The storage

dumpPeriod - The dumpPeriod if any, null otherwise

storageTableDesc - The storage table description

Throws:

LensException

addPartition

public List<org.apache.hadoop.hive.ql.metadata.Partition> addPartition(StoragePartitionDesc partSpec,
                                                                       String storageName,
                                                                       CubeTableType type)
                                                                throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                                                       LensException

Add a partition specified by the storage partition desc on the storage passed.

TODO: separate methods for fact and dim partition addition.

Parameters:

partSpec - The storage partition description

storageName - The storage object

type -

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

addPartitions

public List<org.apache.hadoop.hive.ql.metadata.Partition> addPartitions(List<StoragePartitionDesc> storagePartitionDescs,
                                                                        String storageName,
                                                                        CubeTableType type)
                                                                 throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                                                        LensException

batch addition

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

getStorageTableStartDate

public Date getStorageTableStartDate(String storageTable,
                                     FactTable factTableName)
                              throws LensException

Throws:

LensException

getStorageTableEndDate

public Date getStorageTableEndDate(String storageTable,
                                   FactTable factTableName)
                            throws LensException

Throws:

LensException

dropPartition

public void dropPartition(String cubeTableName,
                          String storageName,
                          Map<String,Date> timePartSpec,
                          Map<String,String> nonTimePartSpec,
                          UpdatePeriod updatePeriod)
                   throws org.apache.hadoop.hive.ql.metadata.HiveException,
                          LensException

Add a partition specified by the storage partition desc on the storage passed.

Parameters:

cubeTableName - cube fact/dimension table name

storageName - storage name

timePartSpec - time partitions

nonTimePartSpec - non time partitions

updatePeriod - update period of the partition

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

tableExists

public boolean tableExists(String tblName)
                    throws org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

factPartitionExists

public boolean factPartitionExists(FactTable fact,
                                   FactPartition part,
                                   String storageTableName)
                            throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                   LensException

extract storage name and check in timeline cache for existance

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

factPartitionExists

public boolean factPartitionExists(String factName,
                                   String storageName,
                                   UpdatePeriod updatePeriod,
                                   Map<String,Date> partitionTimestamp,
                                   Map<String,String> partSpec)
                            throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                   LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

partitionExists

public boolean partitionExists(String storageTableName,
                               UpdatePeriod updatePeriod,
                               Map<String,Date> partitionTimestamps)
                        throws org.apache.hadoop.hive.ql.metadata.HiveException,
                               LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

partitionExistsByFilter

public boolean partitionExistsByFilter(String cubeTableName,
                                       String storageName,
                                       UpdatePeriod updatePeriod,
                                       String filter)
                                throws LensException

Throws:

LensException

partitionExistsByFilter

public boolean partitionExistsByFilter(String storageTableName,
                                       String filter)
                                throws LensException

Throws:

LensException

getAllParts

public List<org.apache.hadoop.hive.ql.metadata.Partition> getAllParts(String storageTableName)
                                                               throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                                                      LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

getPartitionByFilter

public org.apache.hadoop.hive.ql.metadata.Partition getPartitionByFilter(String storageTableName,
                                                                         String filter)
                                                                  throws org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

getPartitionsByFilter

public List<org.apache.hadoop.hive.ql.metadata.Partition> getPartitionsByFilter(String storageTableName,
                                                                                String filter)
                                                                         throws org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

dimTableLatestPartitionExists

public boolean dimTableLatestPartitionExists(String storageTableName)

getHiveTable

public org.apache.hadoop.hive.ql.metadata.Table getHiveTable(String tableName)
                                                      throws LensException

Get the hive Table corresponding to the name

Parameters:

tableName - table name

Returns:

Table object corresponding to the name

Throws:

LensException

getTimePartColNamesOfTable

public List<String> getTimePartColNamesOfTable(String storageTableName)
                                        throws LensException

Throws:

LensException

getTimePartColNamesOfTable

public List<String> getTimePartColNamesOfTable(org.apache.hadoop.hive.ql.metadata.Table table)

extract from table properties

getTableWithTypeFailFast

public org.apache.hadoop.hive.ql.metadata.Table getTableWithTypeFailFast(String tableName,
                                                                         CubeTableType type)
                                                                  throws LensException

Throws:

LensException

getTableWithType

public org.apache.hadoop.hive.ql.metadata.Table getTableWithType(String tableName,
                                                                 CubeTableType type,
                                                                 boolean throwException)
                                                          throws LensException

Throws:

LensException

getTable

public org.apache.hadoop.hive.ql.metadata.Table getTable(String tableName)
                                                  throws LensException

Throws:

LensException

getTable

public org.apache.hadoop.hive.ql.metadata.Table getTable(String tableName,
                                                         boolean throwException)
                                                  throws LensException

Throws:

LensException

dropHiveTable

public void dropHiveTable(String table)
                   throws LensException

Throws:

LensException

isFactTable

public boolean isFactTable(String tableName)
                    throws LensException

Is the table name passed a fact table?

Parameters:

tableName - table name

Returns:

true if it is cube fact, false otherwise

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

isVirtualFactTable

public boolean isVirtualFactTable(String virtualTableName)
                           throws LensException

Is the table name passed a virtual fact table?

Parameters:

virtualTableName - table name

Returns:

true if it is virtual fact, false otherwise

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

isDimensionTable

public boolean isDimensionTable(String tableName)
                         throws LensException

Is the table name passed a dimension table?

Parameters:

tableName - table name

Returns:

true if it is cube dimension, false otherwise

Throws:

LensException

isCube

public boolean isCube(String tableName)
               throws LensException

Is the table name passed a cube?

Parameters:

tableName - table name

Returns:

true if it is cube, false otherwise

Throws:

LensException

isDimension

public boolean isDimension(String tableName)
                    throws LensException

Is the table name passed a dimension?

Parameters:

tableName - table name

Returns:

true if it is dimension, false otherwise

Throws:

LensException

getXFactTable

public XFact getXFactTable(String tableName)
                    throws LensException

Throws:

LensException

getXFactTable

public XFact getXFactTable(FactTable ft)
                    throws LensException

Throws:

LensException

getSegmentationTable

public Segmentation getSegmentationTable(String tableName)
                                  throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                         LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

getXDimensionTable

public XDimensionTable getXDimensionTable(String dimTable)
                                   throws LensException

Throws:

LensException

getXDimensionTable

public XDimensionTable getXDimensionTable(CubeDimensionTable dimTable)
                                   throws LensException

Throws:

LensException

getDimensionTable

public CubeDimensionTable getDimensionTable(String tableName)
                                     throws LensException

Get CubeDimensionTable object corresponding to the name

Parameters:

tableName - The cube dimension name

Returns:

Returns CubeDimensionTable if table name passed is a dimension table

Throws:

LensException - if there is no dimension table with the name

getStorage

public Storage getStorage(String storageName)
                   throws LensException

Get Storage object corresponding to the name

Parameters:

storageName - The storage name

Returns:

Returns storage if name passed is a storage

Throws:

LensException - if there is no storage by the name

getStorage

public Storage getStorage(String storageName,
                          boolean throwException)
                   throws LensException

Throws:

LensException

getCube

public CubeInterface getCube(String tableName)
                      throws LensException

Get Cube object corresponding to the name

Parameters:

tableName - The cube name

Returns:

Returns cube is table name passed is a cube

Throws:

LensException - when the table name does not correspond to a cube

getDimension

public Dimension getDimension(String tableName)
                       throws LensException

Get Cube object corresponding to the name

Parameters:

tableName - The cube name

Returns:

Returns Dimension if table name passed is a Dimension

Throws:

LensException - if the table name passed is not a Dimension

getFactTable

public FactTable getFactTable(String tableName)
                       throws LensException

Get Cube object corresponding to the name

Parameters:

tableName - The cube name

Returns:

Returns cube is table name passed is a cube

Throws:

LensException - if there is no cube by the name

getSegmentation

public Segmentation getSegmentation(String segName)
                             throws LensException

Throws:

LensException

getSegmentation

public Segmentation getSegmentation(String segName,
                                    boolean throwException)
                             throws LensException

Throws:

LensException

getAllDimensionTables

public Collection<CubeDimensionTable> getAllDimensionTables()
                                                     throws LensException

Get all dimension tables in metastore

Returns:

List of dimension tables

Throws:

LensException

getAllStorages

public Collection<Storage> getAllStorages()
                                   throws LensException

Get all storages in metastore

Returns:

List of Storage objects

Throws:

LensException

getAllCubes

public Collection<CubeInterface> getAllCubes()
                                      throws LensException

Get all cubes in metastore

Returns:

List of Cube objects

Throws:

LensException

getAllDimensions

public Collection<Dimension> getAllDimensions()
                                       throws LensException

Get all cubes in metastore

Returns:

List of Cube objects

Throws:

LensException

getAllFacts

public Collection<FactTable> getAllFacts()
                                  throws LensException

Get all facts in metastore

Returns:

List of Cube Fact Table objects

Throws:

LensException

getAllFacts

public Collection<FactTable> getAllFacts(boolean includeVirtualFacts)
                                  throws LensException

Get all facts in metastore (virtual facts optional)

Parameters:

includeVirtualFacts - set true for including virtual facts

Returns:

List of Cube Fact Table objects

Throws:

LensException

getAllSegmentations

public Collection<Segmentation> getAllSegmentations()
                                             throws LensException

Get all segmentations in metastore

Returns:

List of segmentation objects

Throws:

LensException

getAllFacts

public List<FactTable> getAllFacts(CubeInterface cube)
                            throws LensException

Get all fact tables of the cube.

Parameters:

cube - Cube object

Returns:

List of fact tables

Throws:

LensException

getAllFacts

public List<FactTable> getAllFacts(CubeInterface cube,
                                   boolean includeVirtualFacts)
                            throws LensException

Get all facts of cube (optional virtual facts)

Parameters:

cube - Cube object

includeVirtualFacts - set true for virtual facts

Returns:

List of fact tables with optional virtual facts

Throws:

LensException

getAllSegmentations

public List<Segmentation> getAllSegmentations(CubeInterface cube)
                                       throws LensException

Throws:

LensException

getAllDerivedQueryableCubes

public List<DerivedCube> getAllDerivedQueryableCubes(CubeInterface cube)
                                              throws LensException

Get all derived cubes of the cube, that have all fields queryable together

Parameters:

cube - Cube object

Returns:

List of DerivedCube objects

Throws:

LensException

getAllDimensionTables

public List<CubeDimensionTable> getAllDimensionTables(Dimension dim)
                                               throws LensException

Get all dimension tables of the dimension.

Parameters:

dim - Dimension object

Returns:

List of fact tables

Throws:

LensException

partColExists

public boolean partColExists(FactTable factTable,
                             String storage,
                             String partCol)
                      throws LensException

Throws:

LensException

getStorageTables

public Set<String> getStorageTables(FactTable factTable,
                                    String storage)
                             throws LensException

Returns storage table names for a storage. Note: If each update period in the storage has a different storage table, this method will return N Storage Tables where N is the number of update periods in the storage (LENS-1386)

Parameters:

factTable -

storage -

Returns:

Throws:

LensException

pushHiveTable

public void pushHiveTable(org.apache.hadoop.hive.ql.metadata.Table hiveTable)
                   throws org.apache.hadoop.hive.ql.metadata.HiveException,
                          LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterHiveTable

public void alterHiveTable(String table,
                           org.apache.hadoop.hive.ql.metadata.Table hiveTable)
                    throws org.apache.hadoop.hive.ql.metadata.HiveException,
                           LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterCube

public void alterCube(XCube cube)
               throws org.apache.hadoop.hive.ql.metadata.HiveException,
                      LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterCube

public void alterCube(String cubeName,
                      CubeInterface cube)
               throws org.apache.hadoop.hive.ql.metadata.HiveException,
                      LensException

Alter cube specified by the name to new definition

Parameters:

cubeName - The cube name to be altered

cube - The new cube definition Cube or DerivedCube

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterDimension

public void alterDimension(XDimension newDim)
                    throws org.apache.hadoop.hive.ql.metadata.HiveException,
                           LensException

Alter dimension specified by the dimension name to new definition

Parameters:

newDim - The new dimension definition

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterDimension

public void alterDimension(String dimName,
                           Dimension newDim)
                    throws org.apache.hadoop.hive.ql.metadata.HiveException,
                           LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterStorage

public void alterStorage(XStorage storage)
                  throws LensException,
                         org.apache.hadoop.hive.ql.metadata.HiveException

Alter storage specified by the name to new definition

Parameters:

storage - The new storage definition

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

alterStorage

public void alterStorage(String storageName,
                         Storage storage)
                  throws LensException,
                         org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

dropStorage

public void dropStorage(String storageName)
                 throws LensException

Drop a storage

Parameters:

storageName - storage name

Throws:

LensException

dropCube

public void dropCube(String cubeName)
              throws LensException

Drop a cube

Parameters:

cubeName - cube name

Throws:

LensException

dropDimension

public void dropDimension(String dimName)
                   throws LensException

Drop a dimension

Parameters:

dimName - dimension name to be dropped

Throws:

LensException

dropFact

public void dropFact(String factName,
                     boolean cascade)
              throws LensException

Drop a fact with cascade flag

Parameters:

factName - fact name

cascade - If true, will drop all the storages of the fact

Throws:

LensException

dropVirtualFact

public void dropVirtualFact(String virtualFactName)
                     throws LensException

Drop a virtual fact

Parameters:

virtualFactName - virtual fact name

Throws:

LensException

dropSegmentation

public void dropSegmentation(String segName)
                      throws LensException

Throws:

LensException

dropStorageFromFact

public void dropStorageFromFact(String factName,
                                String storage)
                         throws LensException

Drop a storage from fact

Parameters:

factName - fact name

storage - storage name

Throws:

LensException

dropStorageFromDim

public void dropStorageFromDim(String dimTblName,
                               String storage)
                        throws org.apache.hadoop.hive.ql.metadata.HiveException,
                               LensException

Drop a storage from dimension

Parameters:

dimTblName - dim table name

storage - storage

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

dropDimensionTable

public void dropDimensionTable(String dimTblName,
                               boolean cascade)
                        throws LensException

Drop the dimension table

Parameters:

dimTblName - dim table name

cascade - If true, will drop all the dimension storages

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterCubeFactTable

public void alterCubeFactTable(XFact fact)
                        throws LensException,
                               org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

alterCubeFactTable

public void alterCubeFactTable(String factTableName,
                               FactTable cubeFactTable,
                               Map<String,StorageTableDesc> storageTableDescs,
                               Map<String,String> props)
                        throws org.apache.hadoop.hive.ql.metadata.HiveException,
                               LensException

Alter a cubefact with new definition and alter underlying storage tables as well.

Parameters:

factTableName - fact table name

cubeFactTable - cube fact table

storageTableDescs - storage table desc objects

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterSegmentation

public void alterSegmentation(XSegmentation cubeSeg)
                       throws LensException,
                              org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

alterVirtualCubeFactTable

public void alterVirtualCubeFactTable(CubeVirtualFactTable cubeVirtualFactTable)
                               throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                      LensException

Alter a virtual cube fact with new definition

Parameters:

cubeVirtualFactTable - cube virtual fact table

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

alterSegmentation

public void alterSegmentation(String segName,
                              Segmentation seg)
                       throws org.apache.hadoop.hive.ql.metadata.HiveException,
                              LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

getCubeFactTable

public CubeFactTable getCubeFactTable(String factName)
                               throws LensException

Throws:

LensException

getCubeVirtualFactTable

public CubeVirtualFactTable getCubeVirtualFactTable(String factName)
                                             throws LensException

Throws:

LensException

alterCubeDimensionTable

public void alterCubeDimensionTable(XDimensionTable dimensionTable)
                             throws LensException,
                                    org.apache.hadoop.hive.ql.metadata.HiveException

Throws:

LensException

org.apache.hadoop.hive.ql.metadata.HiveException

alterCubeDimensionTable

public void alterCubeDimensionTable(String dimTableName,
                                    CubeDimensionTable cubeDimensionTable,
                                    Map<String,StorageTableDesc> storageTableDescs)
                             throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                    LensException

Alter dimension table with new dimension definition and underlying storage tables as well

Parameters:

dimTableName - dim table name

cubeDimensionTable - cube dimention table

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

isStorageTableCandidateForRange

public boolean isStorageTableCandidateForRange(String storageTableName,
                                               Date fromDate,
                                               Date toDate)
                                        throws LensException

Throws:

LensException

isStorageTablePartitionACandidate

public boolean isStorageTablePartitionACandidate(String storageTableName,
                                                 Date partDate)
                                          throws LensException

Throws:

LensException

isStorageTableCandidateForRange

public boolean isStorageTableCandidateForRange(String storageTableName,
                                               String fromDate,
                                               String toDate)
                                        throws org.apache.hadoop.hive.ql.metadata.HiveException,
                                               LensException

Throws:

org.apache.hadoop.hive.ql.metadata.HiveException

LensException

getStorageTableName

public String getStorageTableName(String factOrDimTableName,
                                  String storage,
                                  UpdatePeriod updatePeriod)
                           throws LensException

Throws:

LensException