ABEJA Datalake documentation

ABEJA Datalake library is SDK for python, which allows developers to create, get and delete datalake resources.

High level API

High level API is used following classes.

• Client
  • Client
    • Client.channels
    • Client.get_channel()
  • Channel
    • Channel
      • Channel.add_datasource()
      • Channel.files
      • Channel.get_file()
      • Channel.list_datasources()
      • Channel.list_files()
      • Channel.remove_datasource()
      • Channel.upload()
      • Channel.upload_dir()
      • Channel.upload_file()
  • Channels
    • Channels
      • Channels.create()
      • Channels.get()
      • Channels.list()
      • Channels.patch()
  • File
    • DatalakeFile
      • DatalakeFile.LIFETIME
      • DatalakeFile.commit()
      • DatalakeFile.get_content()
      • DatalakeFile.get_file_info()
      • DatalakeFile.get_iter_content()
      • DatalakeFile.get_iter_lines()
      • DatalakeFile.get_json()
      • DatalakeFile.get_text()
      • DatalakeFile.lifetime
      • DatalakeFile.to_source_data()

Usage

from abeja.datalake import Client

client = Client()

Low level API

Low level APIs are directly access the API endpoint.

• API Client
  • APIClient
    • APIClient.archive_bucket()
    • APIClient.archive_channel()
    • APIClient.create_bucket()
    • APIClient.create_channel()
    • APIClient.delete_channel_datasource()
    • APIClient.delete_channel_file()
    • APIClient.get_bucket()
    • APIClient.get_bucket_file()
    • APIClient.get_channel()
    • APIClient.get_channel_file_download()
    • APIClient.get_channel_file_upload()
    • APIClient.list_bucket_files()
    • APIClient.list_buckets()
    • APIClient.list_channel_datasources()
    • APIClient.list_channel_files()
    • APIClient.list_channels()
    • APIClient.patch_bucket()
    • APIClient.patch_channel()
    • APIClient.post_channel_file_upload()
    • APIClient.put_channel_datasource()
    • APIClient.put_channel_file_lifetime()
    • APIClient.put_channel_file_metadata()
    • APIClient.unarchive_bucket()
    • APIClient.unarchive_channel()
    • APIClient.upload_bucket_file()
    • APIClient.upload_bucket_files()

Usage

from abeja.datalake import APIClient

api = APIClient()
channel = api.get_channel(organization_id, channel_id)

API Mapping

method	path	description
post	/organizations/<organization_id>/channels	APIClient.create_channel()
get	/organizations/<organization_id>/channels	APIClient.list_channels()
get	/organizations/<organization_id>/channels/<channel_id>	APIClient.get_channel()
patch	/organizations/<organization_id>/channels/<channel_id>	APIClient.patch_channel()
delete	/organizations/<organization_id>/channels/<channel_id>	APIClient.delete_channel()
get	/organizations/<organization_id>/channels/<channel_id>/datasources	APIClient.list_channel_datasources()
put	/organizations/<organization_id>/channels/<channel_id>/datasources/<datasource_id>	APIClient.put_channel_datasource()
delete	/organizations/<organization_id>/channels/<channel_id>/datasources	APIClient.delete_channel_datasource()
post	/channels/	APIClient.get_channel_file_upload()
get	/channels/<channel_id>/	APIClient.list_channel_files()
get	/channels/<channel_id>/<file_id>	APIClient.get_channel_file_download()
delete	/channels/<channel_id>/<file_id>	APIClient.delete_channel_file()
post	/organizations/<organization_id>/buckets	APIClient.create_bucket()
get	/organizations/<organization_id>/buckets	APIClient.list_buckets()
get	/organizations/<organization_id>/buckets/<bucket_id>	APIClient.get_bucket()
patch	/organizations/<organization_id>/buckets/<bucket_id>	APIClient.patch_bucket()
post	/organizations/<organization_id>/buckets/<bucket_id>/archive	APIClient.archive_bucket()
post	/organizations/<organization_id>/buckets/<bucket_id>/unarchive	APIClient.unarchive_bucket()
post	/organizations/<organization_id>/buckets/<bucket_id>/files	APIClient.upload_bucket_file()
post	/organizations/<organization_id>/buckets/<bucket_id>/files	APIClient.upload_bucket_files()
get	/organizations/<organization_id>/buckets/<bucket_id>/files	APIClient.list_bucket_files()
get	/organizations/<organization_id>/buckets/<bucket_id>/files/<file_id>	APIClient.get_bucket_file()

Tutorial

• A Sample tutorial
  • Create a datalake channel and upload files to the channel.
    • Steps 1: Create a Datalake channel
    • Steps 2: Upload file to the Datalake channel
    • Steps 3: Get files from the Datalake channel
  • Use channel