File Adapter Remote

Source Configuration
Determine the configuration parameters for your File adapter remote source and, if
necessary, use these code samples for creating a remote source using the SQL console.

The parameters available to you during this process may change because they depend on
which options you choose.NoteIf you want to use a DPAgent agent installed on Linux to
connect to the SharePoint site, enable basic authentication on the SharePoint server.

Category Option Description

ConnectionInfo Source Options  Local File System: Specifies that the source is
a file on a local system.

 SharePoint Server: Specifies that the source is

on a SharePoint server.

NoteIt does not matter if Source Options is

set to “Local File System” or “Sharepoint Server”;
the File adapter remote source only displays the file
format definitions (.cfg files) under the DP Agent
local folder that you specify in the remote source. It
never displays files or directories on the Sharepoint
server directly, which is different than an Excel
Adapter + Sharepoint scenario. You need to provide
the path to the source file on the Sharepoint site in
the CFG file, and then create the virtual table in the
file format definitions.

Target Options The path to the folder that you want to access on the
local file system where the DP Agent is deployed.

Remote HDFS: Specifies that the target file is on a

remote HDFS system.
Category Option Description

Root Directory The root directory for your data files. This is used
for security. No remote source can reach beyond
this directory for data files. The root directory must
exist before you can create a remote
source.NoteDo not use a link directory or
directory shortcut for a value in this parameter.

If you are using a shared network directory, enter

the path as follows:

\\<host_name>\<directory>

Directory of the Location where you store your file format definition
file format files. This directory must exist before you can create
definitions a remote source. Include the full path and file
name.NoteDo not use a linked directory or
directory shortcut for a value in this parameter.

Use CDC Set this parameter to “True” if you want to use

realtime change data capture with the File adapter.

NoteThe value of this parameter can be changed

when the remote source is suspended.

ConnectionInfo Host Name The remote URL to connect to the remote HDFS,
> HDFS usually defined in core-site.xml..
Configuration

Port Number Port number to connect to the remote HDFS, usually

defined in core-site.xml.

Target File The location to store copied target files in a remote

Location HDFS. For example, you might want to copy a
target file from a local to a remote HDFS.
Category Option Description

Hadoop User The user of the Hadoop system when logging in.
Name This can also be the owner of the file location where
you want to put the copied target files. Make sure
you have the proper permissions to access this
location.

ConnectionInfo Server URL Enter the URL for the server where the SharePoint
> SharePoint source is located.
Server
If you create a new SharePoint site on the server, be
Configuration
sure to include the name of the site at the end of the
URL. For example, if your server name is
http://<server_name>/ and your new site name
is “site1”, your URL would
be “http://<server_name>/site1”

Local Folder The download folder path. This folder is created

Path under the <DPAgent_root>/ directory
automatically.

The local folder path is the location on the DP

Agent machine's local drive where you want to
place your source data files, which the DP Agent
will download from the SharePoint server

For example, if you enter a value for this parameter

such as download/sharepoint, you create the
following
path: <DPAgent_root>/download/shar
epoint.

Credentials Credentials technical user or secondary user

Mode
Select one of the choices, depending on the purpose
of the remote source you want to create.
 Category Option Description

Credentials > AccessToken A password. An access token protects the files from
AccessTokenE access from different agents. Use this password
ntry when creating a remote source.

Credentials > SharePoint User The domain and user name for the SharePoint
SharePoint (Domain\Userna account.
Login me)

SharePoint The password for the SharePoint account.

Password

The following code samples illustrate how to create a remote source using the SQL
console:

Local file system

Example
Sample Code
CREATE REMOTE SOURCE "MyFileSource" ADAPTER "FileAdapter" AT LOCATION
AGENT "MyAgent"

CONFIGURATION

'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ConnectionProperties>

<PropertyEntry name="rootdir">myrootdir</PropertyEntry>

<PropertyEntry name="fileformatdir">myfileformatdir</PropertyEn
try>

<PropertyEntry name="source_options">local</PropertyEntry>

<PropertyEntry name="target_options">local</PropertyEntry>

</ConnectionProperties>

' WITH CREDENTIAL TYPE 'PASSWORD' USING

'<CredentialEntry name="AccessTokenEntry">

<password>mytoken</password>

</CredentialEntry>';

Local file system with real-time

Example
Sample Code
CREATE REMOTE SOURCE "MyFileSource" ADAPTER "FileAdapter" AT LOCATION
AGENT "MyAgent"

CONFIGURATION

'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ConnectionProperties>

<PropertyEntry name="rootdir">myrootdir</PropertyEntry>

<PropertyEntry name="fileformatdir">myfileformatdir</PropertyEn
try>

<PropertyEntry name="usecdc">true</PropertyEntry>

<PropertyEntry name="source_options">local</PropertyEntry>

<PropertyEntry name="target_options">local</PropertyEntry>

</ConnectionProperties>

' WITH CREDENTIAL TYPE 'PASSWORD' USING

'<CredentialEntry name="AccessTokenEntry">

<password>mytoken</password>

</CredentialEntry>';

HDFS as target

Example
Sample Code
CREATE REMOTE SOURCE "MyFileSource" ADAPTER "FileAdapter" AT LOCATION
AGENT "MyAgent"

CONFIGURATION

'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ConnectionProperties>

<PropertyEntry name="rootdir">myrootdir</PropertyEntry>

<PropertyEntry name="fileformatdir">myfileformatdir</PropertyEn
try>

<PropertyEntry name="target_options">hdfs</PropertyEntry>

<PropertyEntry name="host">hdfs:/myhost.sap.corp</PropertyEntry
>

<PropertyEntry name="port_number">8020</PropertyEntry>

<PropertyEntry name="target_hdfs_file_location">/user/sap</Prop
ertyEntry>

<PropertyEntry name="hadoop_user_name">myuser</PropertyEntry>

</ConnectionProperties>

' WITH CREDENTIAL TYPE 'PASSWORD' USING

'<CredentialEntry name="AccessTokenEntry">

<password>mytoken</password>

</CredentialEntry>';

Sharepoint location

Example
Sample Code
CREATE REMOTE SOURCE "MyFileSource" ADAPTER "FileAdapter" AT LOCATION
AGENT "MyAgent"

CONFIGURATION

'<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ConnectionProperties>

<PropertyEntry name="rootdir">myrootdir</PropertyEntry>

<PropertyEntry name="fileformatdir">myfileformatdir</PropertyEn
try>

<PropertyEntry name="source_options">SharePoint</PropertyEntry>

<PropertyEntry name="target_options">local</PropertyEntry>

<PropertyGroup name="SharePoint">

<PropertyEntry name="spurl">http://myhost.sap.corp/mysharepoint
site</PropertyEntry>

<PropertyEntry name="spdir">download/sharepoint</PropertyEntry>

</PropertyGroup>

</ConnectionProperties>

' WITH CREDENTIAL TYPE 'PASSWORD' USING

'<CredentialEntry name="AccessTokenEntry">

<password>mytoken</password>

</CredentialEntry>

<CredentialEntry name="sharePointCredential">

<user>mydomain\mysharepointuser</user>

<password>mypassword</password>

</CredentialEntry>';