Page History

...

Help for interacting with the swift API using HTTP requests

1. Create token in OpenStack using this script.

   Code Block

   export OS_AUTH_URL=https://keystone.cloud.ifca.es:5000/v3 # With the addition of Keystone we have standardized on the term **tenant** # as the entity that owns the resources. export OS_TENANT_ID=dfd23dc772fe4f76ad34d0013e44a745 export OS_TENANT_NAME=ifca.es:mods export OS_USER_DOMAIN_NAME=IFCA export OS_IDENTITY_API_VERSION=3 # In addition to the owning entity (tenant), openstack stores the entity # performing the action as the **user**. export OS_USERNAME=aidaph # With Keystone you pass the keystone password. read -s -p Password: pwd export OS_PASSWORD=$pwd openstack --os-username=$OS_USERNAME --os-user-domain-name=$OS_USER_DOMAIN_NAME --os-password="$OS_PASSWORD" --os-project-name=$OS_TENANT_NAME --os-project-name=$OS_USER_DOMAIN_NAME token issue

   
   

2. Object Storage API 

   1. Show object metadata using the HEAD sentence

      Code Block

      @nova:~$ curl -s --head -H "X-Auth-Token: $TOKEN " https://api.cloud.ifca.es:8080/swift/v1/SparkTest/country_vaccinations.csv HTTP/1.1 200 OK Content-Length: 1075352 Accept-Ranges: bytes Last-Modified: Wed, 30 Mar 2022 10:17:13 GMT X-Timestamp: 1648635433.45941 etag: 391086e2094ca2c92106fd1cf39765b6 X-Object-Meta-Orig-Filename: country_vaccinations.csv X-Trans-Id: tx000000000000000000f4c-006246d490-44232c6-RegionOne X-Openstack-Request-Id: tx000000000000000000f4c-006246d490-44232c6-RegionOne Content-Type: text/csv Date: Fri, 01 Apr 2022 10:31:45 GMT

      
      

   2. create or update an object or its metadata

   1. using POST

      Code Block

      @nova:~$ curl -X POST -H "X-Auth-Token: $TOKEN" -H "X-Object-Meta-Orig-Filename: contry-vaccinations-test.csv" https://api.cloud.ifca.es:8080/swift/v1/SparkTest/country_vaccinations.csv @nova:~$ curl -s --head -H "X-Auth-Token: $TOKEN " https://api.cloud.ifca.es:8080/swift/v1/SparkTest/country_vaccinations.csv HTTP/1.1 200 OK Content-Length: 1075352 X-Timestamp: 1648810529.55442 X-Object-Meta-Orig-Filename: contry-vaccinations-test.csv [...] Date: Fri, 01 Apr 2022 10:55:35 GMT

      To create or update custom metadata, use the X-Object-Meta-name header, where name is the name of the metadata item. The X-Object-Meta-Orig-Filename is an example of metadata  field where Orig-Filename is the name of the metadata object. In addition to the custom metadata, you can update the Content-Type, Content-Encoding, Content-Disposition, and X-Delete-At system metadata items. However you cannot update other system metadata, such as Content-Length or Last-Modified.

Ceph Object Gateway Swift integration with Delta Lake and Apache Hadoop/Spark

For using the Delta Lake storage under Ceph Rados Gateway, an implementation of ceph rados gw for delta using OpenStack Swift API will be included in the next releases of Delta. This code is officially under testing, and we check if it works at IFCA.

The test can be launched though a Spark interactive session. First of all, the dependencies must be download before starting the session. These includes the Delta-core and the implementation of ceph with hadoop called  hadoop-cephrgw. Apart from that, some variables are mandatory in the hadoop configuration. The variables can be set in the core-site.xml inside the etc/hadoop path of your current $HOME of Hadoop or you can pass it directly from the terminal. This is an example of the command:

hadoopuser@mods-hadoop:~$ export _JAVA_OPTIONS='-Djdk.tls.maxCertificateChainLength=15'; \ 
spark-shell --packages io.delta:delta-core_2.12:1.1.0,io.github.nanhu-lab:hadoop-cephrgw:1.0.1,org.apache.spark:spark-hadoop-cloud_2.12:3.2.1\
 --conf spark.hadoop.fs.ceph.username=tester \
 --conf spark.hadoop.fs.ceph.password=testXXX \
 --conf spark.hadoop.fs.ceph.uri=https://api.cloud.ifca.es:8080/swift/v1/ 
 --conf spark.hadoop.fs.s3a.connection.ssl.enabled=false 
 --conf spark.delta.logStore.class=org.apache.spark.sql.delta.storage.S3SingleDriverLogStore 
 --conf spark.hadoop.fs.ceph.impl=org.apache.hadoop.fs.ceph.rgw.CephStoreSystem

S3 Configuration

TODO

...