org.locationtech.geowave.format.avro

Class GeoWaveAvroIngestPlugin

java.lang.Object

org.locationtech.geowave.adapter.vector.ingest.AbstractSimpleFeatureIngestPlugin<AvroSimpleFeatureCollection>

org.locationtech.geowave.format.avro.GeoWaveAvroIngestPlugin

All Implemented Interfaces:

Persistable, GeoWaveAvroFormatPlugin<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature>, GeoWaveAvroPluginBase<AvroSimpleFeatureCollection>, GeoWaveAvroSchemaProvider, IngestFromHdfsPlugin<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature>, DataAdapterProvider<org.opengis.feature.simple.SimpleFeature>, IndexProvider, IngestPluginBase<URL,org.opengis.feature.simple.SimpleFeature>, LocalFileIngestPlugin<org.opengis.feature.simple.SimpleFeature>, LocalPluginBase

public class GeoWaveAvroIngestPlugin
extends AbstractSimpleFeatureIngestPlugin<AvroSimpleFeatureCollection>

This plugin is used for ingesting any GPX formatted data from a local file system into GeoWave as GeoTools' SimpleFeatures. It supports the default configuration of spatial and spatial-temporal indices and it will support wither directly ingesting GPX data from a local file system to GeoWave or to stage the data in an intermediate format in HDFS and then to ingest it into GeoWave using a map-reduce job. It supports OSM metadata.xml files if the file is directly in the root base directory that is passed in command-line to the ingest framework.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	GeoWaveAvroIngestPlugin.IngestAvroFeaturesFromHdfs

Nested classes/interfaces inherited from class org.locationtech.geowave.adapter.vector.ingest.AbstractSimpleFeatureIngestPlugin

AbstractSimpleFeatureIngestPlugin.AbstractIngestSimpleFeatureWithMapper<I>

Field Summary

Fields inherited from class org.locationtech.geowave.adapter.vector.ingest.AbstractSimpleFeatureIngestPlugin

filterOptionProvider, serializationFormatOptionProvider, simpOptionProvider, typeNameProvider

Constructor Summary

Constructors

Constructor and Description
GeoWaveAvroIngestPlugin()

Method Summary

Modifier and Type	Method and Description
org.apache.avro.Schema	getAvroSchema() Returns the Avro schema for the plugin
String[]	getFileExtensionFilters() Gets a list of file extensions that this plugin supports.
IngestPluginBase<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature>	getIngestWithAvroPlugin() An implementation of ingestion that ingests Avro Java objects into GeoWave
Index[]	getRequiredIndices() Get an array of indices that are required by this ingest implementation.
Class<? extends CommonIndexValue>[]	getSupportedIndexableTypes() return a set of classes that can be indexed by this data adapter provider, used for compatibility checking with an index provider
protected org.opengis.feature.simple.SimpleFeatureType[]	getTypes()
IngestWithMapper<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature>	ingestWithMapper() An implementation of ingestion that can be persisted to a mapper within the map-reduce job configuration to perform an ingest of data into GeoWave from intermediate data
IngestWithReducer<AvroSimpleFeatureCollection,?,?,org.opengis.feature.simple.SimpleFeature>	ingestWithReducer() An implementation of ingestion that can be persisted to a mapper and reducer within the map-reduce job configuration to aggregate intermediate data by defined keys within a reducer and perform an ingest of data into GeoWave from the key-value pairs emitted by the mapper.
void	init(URL baseDirectory) Initialize the plugin and give it the base directory that is provided by the user.
boolean	isUseReducerPreferred() Returns a flag indicating to the ingestion framework whether it should try to use the ingestWithMapper() implementation or the ingestWithReducer() implementation in the case that both implementations are non-null.
boolean	supportsFile(URL file) This method will be called for every file that matches the given extensions.
CloseableIterator<AvroSimpleFeatureCollection>	toAvroObjects(URL input) Converts the supported file into an Avro encoded Java object.
protected CloseableIterator<GeoWaveData<org.opengis.feature.simple.SimpleFeature>>	toGeoWaveDataInternal(AvroSimpleFeatureCollection featureCollection, String[] indexNames, String globalVisibility)

Methods inherited from class org.locationtech.geowave.adapter.vector.ingest.AbstractSimpleFeatureIngestPlugin

fromBinary, getDataAdapters, newAdapter, setFilterProvider, setGeometrySimpOptionProvider, setSerializationFormatProvider, setTypeNameProvider, toBinary, toGeoWaveData, wrapIteratorWithFilters

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GeoWaveAvroIngestPlugin

public GeoWaveAvroIngestPlugin()

Method Detail

getFileExtensionFilters

public String[] getFileExtensionFilters()

Description copied from interface: LocalPluginBase

Gets a list of file extensions that this plugin supports. If not provided, this plugin will accept all file extensions.

Returns:

The array of file extensions supported ('.' is unnecessary)

init

public void init(URL baseDirectory)

Description copied from interface: LocalPluginBase

Initialize the plugin and give it the base directory that is provided by the user.

Parameters:

baseDirectory - The base directory provided as a command-line argument (if the argument is a file, the base directory given will be its parent directory).

supportsFile

public boolean supportsFile(URL file)

Description copied from interface: LocalPluginBase

This method will be called for every file that matches the given extensions. It is an opportunity for the plugin to perform arbitrarily complex acceptance filtering on a per file basis, but it is important to understand performance implications if the acceptance test is too intensive and the directory of files to recurse is large.

Parameters:

file - The file to determine if this plugin supports for ingestion

Returns:

Whether the file is supported or not

getTypes

protected org.opengis.feature.simple.SimpleFeatureType[] getTypes()

Specified by:

getTypes in class AbstractSimpleFeatureIngestPlugin<AvroSimpleFeatureCollection>

getAvroSchema

public org.apache.avro.Schema getAvroSchema()

Description copied from interface: GeoWaveAvroSchemaProvider

Returns the Avro schema for the plugin

Returns:

the Avro schema for the intermediate data

toAvroObjects

public CloseableIterator<AvroSimpleFeatureCollection> toAvroObjects(URL input)

Description copied from interface: GeoWaveAvroPluginBase

Converts the supported file into an Avro encoded Java object.

Parameters:

input - The file to convert to Avro

Returns:

The Avro encoded Java object

isUseReducerPreferred

public boolean isUseReducerPreferred()

Description copied from interface: IngestFromHdfsPlugin

Returns a flag indicating to the ingestion framework whether it should try to use the ingestWithMapper() implementation or the ingestWithReducer() implementation in the case that both implementations are non-null.

Returns:

If true, the framework will use ingestWithReducer() and only fall back to ingestWithMapper() if necessary, otherwise the behavior will be the reverse

ingestWithMapper

public IngestWithMapper<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature> ingestWithMapper()

Description copied from interface: IngestFromHdfsPlugin

An implementation of ingestion that can be persisted to a mapper within the map-reduce job configuration to perform an ingest of data into GeoWave from intermediate data

Returns:

The implementation for ingestion with only a mapper

ingestWithReducer

public IngestWithReducer<AvroSimpleFeatureCollection,?,?,org.opengis.feature.simple.SimpleFeature> ingestWithReducer()

Description copied from interface: IngestFromHdfsPlugin

An implementation of ingestion that can be persisted to a mapper and reducer within the map-reduce job configuration to aggregate intermediate data by defined keys within a reducer and perform an ingest of data into GeoWave from the key-value pairs emitted by the mapper.

Returns:

The implementation for ingestion with a mapper and reducer. It is important to provide the correct concrete implementation of Key and Value classes within the appropriate generics because the framework will use reflection to set the key and value classes for map-reduce.

toGeoWaveDataInternal

protected CloseableIterator<GeoWaveData<org.opengis.feature.simple.SimpleFeature>> toGeoWaveDataInternal(AvroSimpleFeatureCollection featureCollection,
                                                                                                         String[] indexNames,
                                                                                                         String globalVisibility)

Specified by:

toGeoWaveDataInternal in class AbstractSimpleFeatureIngestPlugin<AvroSimpleFeatureCollection>

getRequiredIndices

public Index[] getRequiredIndices()

Description copied from interface: IndexProvider

Get an array of indices that are required by this ingest implementation. This should be a subset of supported indices. All of these indices will automatically be persisted with GeoWave's metadata store (and in the job configuration if run as a job), whereas indices that are just "supported" will not automatically be persisted (only if they are the primary index). This is primarily useful if there is a supplemental index required by the ingest process that is not the primary index.

Returns:

the array of indices that are supported by this ingest implementation

getIngestWithAvroPlugin

public IngestPluginBase<AvroSimpleFeatureCollection,org.opengis.feature.simple.SimpleFeature> getIngestWithAvroPlugin()

Description copied from interface: GeoWaveAvroFormatPlugin

An implementation of ingestion that ingests Avro Java objects into GeoWave

Returns:

The implementation for ingestion from Avro

getSupportedIndexableTypes

public Class<? extends CommonIndexValue>[] getSupportedIndexableTypes()

Description copied from interface: DataAdapterProvider

return a set of classes that can be indexed by this data adapter provider, used for compatibility checking with an index provider

Returns:

the classes that are indexable by this adapter provider